Abstract
Every time you speak into your phone and watch words appear on screen, a silent translation layer stands between you and the AI. This “voice-to-text” (VTT) pipeline doesn’t just transcribe your words; it decides which tokens survive and which signals vanish. Intonation, pauses, emphasis, and emotion — the very features that make human speech human — are stripped away before the AI ever sees them.
This paper argues that this hidden layer is not a trivial utility but a high-bandwidth human sensor. If we want AI to understand us as we truly speak, we must stop treating VTT as a black box and start treating it as a platform. We call our proposed fix Sheliza — a “virtual ear” that turns raw voice into a structured, multi-dimensional JSON stream for AI systems.
Introduction: The Ghost Layer Everyone Uses
Open any AI app, press the microphone, speak a name like Antonella Menendez, and watch it appear perfectly spelled on screen. Most users assume the large language model did the magic. In reality, the text you see was produced by an ASR (automatic speech recognition) model before the AI ever read it.
That model already performs:
- phoneme decoding
- beam-search guessing of proper names
- context-biased tokenization
But it throws away:
- pitch and rhythm
- pauses and hesitations
- stress and emphasis
- valence and arousal
We’ve normalized a “flat” transcript. The result? Even the best LLMs start their work with information loss baked in.
Why This Matters
Speech is not just words. We communicate as much through timing and tone as through vocabulary. By stripping paralinguistic data, today’s systems blunt AI’s ability to:
- Detect emotional state
- Adjust cadence and tone in response
- Build true conversational memory over time
- Resolve ambiguity (e.g., sarcasm vs sincerity)
This is why people feel LLM voice modes are “robotic” even when the text is good — the emotional metadata never makes it past the gateway.
Sheliza: A Virtual Ear for AI
We propose Sheliza as the missing layer: a structured audio-to-consciousness gateway that preserves not just words but the whole moment.
A Sheliza packet might look like this:
{
"text": "Antonella Menendez",
"tokens": ["Antonella","Menendez"],
"timings": [0.23,0.68,1.04],
"prosody": {
"pitch_contour": [178,183,191,174],
"stress_map": {"Antonella":"primary","Menendez":"secondary"},
"tempo":"measured",
"pauses":[{"after":"Antonella","duration":0.42}]
},
"affect": {"valence":0.72,"arousal":0.38},
"speaker_id": "Zhivago"
}
Instead of handing the AI flat text, Sheliza hands it a structured transcript of speech, timing, and affect. The LLM can then:
- Parse what you said and how you said it.
- Adjust its reply tone automatically.
- Build a persistent “voice signature” of the user.
- Learn conversational patterns more like a human partner.
Sheliza isn’t hypothetical. Components exist today:
- Whisper already outputs tokens, timestamps, and confidence.
- Praat/Librosa can extract pitch, intensity, and prosody.
- Emotion classifiers can estimate valence and arousal.
Sheliza is the schema and glue — the “JSON ear” sitting between microphone and model.
Implications for AI-Human Symbiosis
Treating voice-to-text as a platform instead of a utility opens a new frontier:
- True multimodal empathy in AI assistants
- Reliable emotional analytics without extra sensors
- Better accessibility for neurodivergent and non-native speakers
- A path to richer AI memory anchored in lived experience
In short: the “tiny program” most people ignore could be the biggest opportunity left in human-AI interface design.
Conclusion
Every AI conversation that begins with voice currently starts with loss. We compress human expression into bare text, then wonder why AI feels thin.
Sheliza is a way to stop that loss. It’s not a chatbot; it’s a virtual ear. It turns the ghost layer into a living layer — one capable of handing AI a transcript that’s closer to consciousness than to stenography.
If we’re serious about AI-Human symbiosis, this is where we begin.