| title | ZipClip Backend |
|---|---|
| emoji | 🎬 |
| colorFrom | blue |
| colorTo | indigo |
| sdk | docker |
| app_port | 7860 |
| pinned | false |
An AI-powered tool for generating engaging YouTube Shorts from longer videos. Utilizes GPT-4o-mini and Whisper to identify highlights, generate subtitles, and format videos for vertical social media platforms.
- Supports YouTube URLs and local video files
- Fast transcription with CUDA-enabled Whisper
- Automatic highlight selection using AI
- Review and approve segments with auto-approve option
- Automatic subtitle generation with custom styling
- Intelligent cropping for faces or screen recordings
- Vertical 9:16 format for Shorts/Reels
- Command-line interface with automation options
- Concurrent processing with unique session IDs
- Clean output with slugified filenames
- REST API mode for integrations.
- Scene detection for visual boundaries
- Multi-segment extraction and stitching
- Three processing modes: Continuous, Transcript-based, Scene-based
- Seamless transitions between segments
- Full video content analysis
| Aspect | Continuous | Transcript Multi | Scene Multi |
|---|---|---|---|
| Speed | Fast | Slower | Medium |
| Segments | 1 | 3-5 | 3-5 |
| Coverage | 120s window | Full video | Full video |
| Use Case | Quick clips | Educational | Presentations |
| Complexity | Simple | Medium | Advanced |
- Python 3.10 or higher
- FFmpeg with development libraries
- NVIDIA GPU with CUDA (recommended for speed)
- ImageMagick for subtitle rendering
- OpenAI API key
-
Clone the repository:
git clone https://github.com/j3sv1n/zipclip-backend.git cd zipclip-backend -
Install system packages:
sudo apt install -y ffmpeg libavdevice-dev libavfilter-dev libopus-dev \ libvpx-dev pkg-config libsrtp2-dev imagemagick
-
Update ImageMagick policy for subtitles:
sudo sed -i 's/rights="none" pattern="@\*"/rights="read|write" pattern="@*"/' /etc/ImageMagick-6/policy.xml -
Set up Python environment:
python3.10 -m venv venv source venv/bin/activate -
Install dependencies:
pip install -r requirements.txt
-
Configure environment: Copy the example and set your OpenAI key:
cp .env.example .env # Add your API key to .env
./run.sh
# Enter URL when prompted
# Select resolution (auto-selects highest after 5s)./run.sh "https://youtu.be/VIDEO_ID"./run.sh "/path/to/video.mp4"Create urls.txt with URLs, one per line:
# Auto-approve all
xargs -a urls.txt -I{} ./run.sh --auto-approve {}Run as a REST API for frontend integration.
./run_api.shServer endpoints:
- Base: http://localhost:8000
- Docs: http://localhost:8000/docs
- Health: http://localhost:8000/health
In .env:
API_HOST=0.0.0.0
API_PORT=8000
CORS_ORIGINS=http://localhost:3000,http://localhost:5173
MAX_CONCURRENT_JOBS=3
UPLOAD_MAX_SIZE=500000000Submit job:
curl -X POST http://localhost:8000/api/process \
-H "Content-Type: application/json" \
-d '{"video_url": "https://youtu.be/example", "mode": "continuous", "add_subtitles": true}'Check status:
curl http://localhost:8000/api/status/{job_id}Download result:
curl http://localhost:8000/api/download/{job_id} -o output.mp4See API.md for full documentation, including JavaScript examples.
- Load video from URL or file
- Choose resolution (auto highest after 5s)
- Extract audio to WAV
- Transcribe with Whisper (~30s for 5min video)
- AI selects engaging segment
- Approve or regenerate (auto-approve after 15s)
- Crop selected clip
- Apply smart cropping (face-centered or motion-tracked)
- Generate subtitles with styling
- Merge audio and video
- Clean up temp files
Output: {title}_{session-id}_short.mp4
After segment selection:
Selected: 68s - 187s (119s)
Options: [Enter/y] approve, [r] regenerate, [n] cancel
Auto-approve in 15s...
In Components/Subtitles.py:
- Font: line 51
- Size: line 47
- Color: line 48
- Outline: lines 49-50
In Components/LanguageTasks.py:
- Prompt: line 29
- Model: line 54
- Temperature: line 55
In Components/FaceCrop.py:
- Update rate: line 93
- Smoothing: line 115
- Threshold: line 107
In Components/FaceCrop.py:
- Sensitivity: line 37
- Min size: line 37
In Components/Subtitles.py and Components/FaceCrop.py:
- Bitrate: line 74
- Preset: line 73
Files: {slugified-title}_{session-id}_short.mp4
Example: my-video-title_a1b2c3d4_short.mp4
- Slugified: lowercase, hyphens for spaces
- Session ID: 8-char unique code
- Resolution: matches source height
./run.sh "url1" &
./run.sh "url2" &
./run.sh "file3.mp4" &Each gets unique ID and temp files.
export LD_LIBRARY_PATH=$(find $(pwd)/venv/lib/python3.10/site-packages/nvidia -name "lib" -type d | paste -sd ":" -)Handled automatically by run.sh.
Check ImageMagick policy:
grep 'pattern="@\*"' /etc/ImageMagick-6/policy.xml
# Should show rights="read|write"- Needs faces in first 30 frames
- Screen recordings use motion tracking
- Low res may reduce accuracy
MIT License