Add streaming support for LLM and TTS models for faster response
1 unresolved thread
Commit also adds unit test basis as logic is getting more complex
Merge request reports
Activity
Filter activity
requested review from @tinjap
assigned to @jari.helaakoski
6 from http.server import BaseHTTPRequestHandler,HTTPServer 7 from os import curdir, sep 8 import torch 9 from diffusers import AutoPipelineForText2Image, LCMScheduler 10 import simplejson 11 import base64 12 from io import BytesIO 13 from PIL import Image 14 15 16 17 PORT_NUMBER = 8005 18 19 #This class will handles any incoming request from 20 #the browser 21 class myHandler(BaseHTTPRequestHandler): changed this line in version 2 of the diff
added 10 commits
- 25c21ee3 - Structured output model
- 144573f2 - Merge branch 'structured-output-model' into 'main'
- 757925c1 - Prototype streaming
- eb8338e1 - Ignore build directory
- 06962f21 - Fixes for streaming proto
- 9ecfbfdc - Add unit test + many fixes
- af4c1f77 - Fixes for tts streaming
- 6d18bf45 - More tests and code fixes. Improve the test app functionality.
- 2341a9c9 - Move servers into separate directory
- c4006fa8 - Fixes
Toggle commit list
Please register or sign in to reply