Building AI Models Faster And Cheaper Than You Think

Coming Up (00:00:00)

• Recent advancements in AI have made many science fiction concepts a reality.
• Generative AI models like GPT-4, Midjourney, and now Sora are pushing the boundaries of what's possible.
• YC companies are building foundation models during the batch with just $500,000.
• These models are being developed by young college graduates in a relatively short time frame.
• This demonstrates that it's possible to be on the cutting edge of AI research without significant resources.

Sora Videos (00:01:13)

• Sora's video showcases a humanoid robot walking a golden retriever on a suburban street.
• The video demonstrates significant improvements in text generation, with the model accurately spelling out "help" and producing high-definition images.
• The physics of the robot's and dog's movements are mostly accurate, capturing the lifelike gait of a golden retriever.
• The prompt was followed precisely, although minor imperfections were noted, such as a floating dog and inconsistencies in the street and structures.
• Sora's videos exhibit long-term visual consistency, maintaining a consistent architectural style and environment throughout the minute-long clip.
• The drone camera circles the Golden Gate Bridge, showcasing stunning views of the cliffs, ocean waves, and San Francisco in the background.
• The high definition of the video is impressive, capturing intricate details of the bridge and city.
• Geographical accuracy is not perfect, with terrain and city layout differing from the real world.
• Minor imperfections include disjointed bridge columns at certain angles and cars driving on the wrong side of the road.
• Simulating fluid motion remains a challenge, resulting in slightly static waves.

How Sora works under the hood? (00:05:05)

• Sora combines a transformer model, typically used for text, with a diffusion model, used in image generation like DALL-E and Midjourney.
• It adds a temporal component to ensure consistency between frames and time.
• Sora is trained with videos and "SpaceTime patches," which are 3x3 matrices of pixels that include spatial and temporal information.
• The size of these patches can vary, and they are trained in a large architecture.
• SpaceTime patches are the video equivalent of tokens, building on prior work in transformer models for images and robotics.

How expensive is it to generate videos vs. texts? (00:08:19)

• Generating videos is more computationally expensive than generating text due to the additional dimension of time.
• GPT-4 has a trillion parameters and operates in two dimensions, while videos require an order of magnitude more parameters, likely around 10 trillion.
• It likely requires 10 times the number of GPUs used for GPT-4, which was around 20,000-30,000 GPUs.
• Some YC companies have achieved similar functionality with fewer resources by optimizing data, compute, and expertise.

Infinity AI (00:10:01)

• Makes deep fake videos of a particular person.
• Trained their model on the first three episodes of the Lite cone podcast.
• Only needed an hour or so of YouTube video to get an accurate representation.

Sync Labs (00:11:23)

• API for creating real-time lip-syncing.
• Trained the models on a single A100 GPU.
• Compressed a lot of the data and used low-resolution video to reduce the amount of data needed.
• Partnered with Aure to get access to a dedicated GPU cluster, allowing them to iterate 100 times faster.
• YC companies get over half a million in credits and instant access to a GPU cluster within 24 hours.
• The companies in the YC batch didn't have to use any of the YC money to train their models.

Sonauto (00:13:41)

• Sonauto is a company that has built a text-to-song model.
• The model can generate songs based on given lyrics and the specified singer.
• The founders of Sonauto are 21 years old and built the model in months by teaching themselves.
• The generated songs have understandable lyrics and sound like they are sung by a person.

Metalware (00:15:44)

• Metalware is a company that is building a co-pilot for hardware design.
• The founders of Metalware had a background in hardware engineering but not in AI.
• They trained a foundation model for hardware design during the batch without much AI expertise.
• Metalware used high-quality data from textbooks and a smaller model (GPT-2.5) to reduce computational resources.
• By constraining tasks, using high-quality data, and choosing a smaller model, Metalware was able to build a foundation model for various applications beyond just generating video or text.

Guide Labs (00:17:40)

• Building an explainable foundation model to understand how the model makes predictions.
• The team is training a model to determine when it's better to invest in building a custom model or fine-tuning an open-source model.
• Expertise in AI might be overrated, as smart individuals who are willing to read research papers can achieve similar results.
• YC can provide credits to offset some of the compute costs.
• The key differentiator lies in finding high-quality data, even if it's not a giant dataset.

Phind (00:19:29)

• Phind is a company that created a co-pilot for software.
• They used synthetic data from programming competitions to train their model.
• Synthetic data was initially controversial because it seemed like a model couldn't generate its own data and learn from it.
• However, it works because LLMs are capable of reasoning, which allows them to generate data and improve their own models.
• Other generative AI models, like self-driving car models, are also trained on massive amounts of simulation data.
• Sora is an AI model that can generate videos.
• It uses video footage generated from game engines like Unreal Engine or Unity, which have full physics simulators.
• This allows Sora to generate videos from multiple camera angles and simulate the real world.
• The implications of this technology go beyond entertainment, as it can be used for weather prediction, scientific simulations, and more.

Diffuse Bio (00:24:21)

• Diffuse Bio applies foundation models to biology to create new molecules for drugs and gene therapies.
• The founder has expertise in biology and published papers in Nature.
• Custom kernels were built to speed up the model training process, reducing resource requirements.

Piramidal (00:25:36)

• Piramidal builds a foundation model for the human brain to predict EEG signals.
• EEG signals are similar to videos, representing electrical impulses over time.
• Chunking the data into spacetime chunks reduced the runtime complexity quadratically.
• The model can be trained with just 800 hours of GPU compute.
• EEG data is an unexpected application area for foundation models.

K-Scale Labs (00:27:15)

• K-Scale Labs is developing consumer humanoid robots.
• The founder previously built the foundation robotics model for Tesla and integrated it into the Optimus Prime robot.
• Advances in foundation models, such as the physics simulator for the world, are enabling breakthroughs in robotics.

DraftAid (00:28:58)

• DraftAid is building AI models for CAD design.
• Traditional CAD software uses old kernels that run on Fortran and are expensive to use.
• DraftAid is using AI models to replace some of these kernels, making the process faster and cheaper.

Playground (00:30:38)

• Playground is a YC company that has developed an AI model that can generate images.
• The model is open-source and outperforms Stable Diffusion in many cases.
• Playground was able to achieve this on far less money than Stability AI and other teams in the space.
• Suil Doshi, the founder of Playground, taught himself AI in a month by reading papers and meeting with experts in the field.
• This highlights the fact that the AI field is still new and that it is possible to become an expert in a relatively short amount of time.
• Companies can compete with OpenAI and other large AI companies by training their own models for specific verticals and use cases.

Outro (00:33:20)

• There are many incredible things being done in AI by people who are likely not that different from the viewers.
• Many notable figures in AI, such as Sam Altman and Dario Amade, started somewhere, and YC could be the starting point for aspiring individuals.