Prompts to perception: The next leap in human-AI interaction

As artificial intelligence advances, the relationship between humans and machines is evolving, moving from simple typed commands and algorithmic responses toward a deeper, more intuitive level of human perception and interaction. This paper explores how the human-AI relationship is shifting from straightforward prompts (“tell me …”, “show me …”) to more complex perceptual interactions — how systems understand context, adjust to human signals, and become collaborators in meaning-making rather than mere tools.

1. The era of prompts: controlling AI through language

Historically, human-AI interaction has been dominated by prompt textual instructions or queries that instruct an AI to perform a task. With the rise of large-language models (LLMs) and generative AI, the prompt became the primary interface: users write or speak a directive, and the model responds.

As Olla et al. observe, “prompts serve as the primary language for communicating with these models” and mark the transition from traditional command interfaces to more open-ended predictive systems.

Prompt engineering has emerged as a discipline focused on crafting concise, precise and layered inputs to elicit desirable outputs. However, this prompt-based approach has limitations: it assumes users know what to ask and that AI can respond correctly. This places the burden of meaning-making largely on humans, often requiring trial and error, refinements and domain expertise.

2. Enter perception: beyond the prompt

The next leap moves from prompts to perception, transitioning from what we say to how the system sees, senses and interprets context, human intent, non-verbal cues, and richer modalities. Research shows humans interacting with AI already treat these systems as social actors. McKee et al. found people attribute warmth and competence to AI systems — judgments shaping their willingness to cooperate.

Similarly, users’ mental models of AI affect engagement: perceiving a system as intentional, independent, or aligned with human interests changes trust and interaction dynamics. This shifts the interface from text-prompting toward a face-to-face-like relationship, raising significant design, ethical and cognitive questions.

Additionally, evidence shows human — AI interaction influences human perception and decision-making: a recent study found biased AI systems create feedback loops, increasing bias in human judgments over time. This indicates that as AI perceives and responds, so does the human side, not just in outputs but in perception, judgment and decision — making.

At Chance AI, we have witnessed firsthand how the shift from prompts to perception reshapes user experience in profound ways. Our app, Chance, is built as the world’s first visual agent — a platform where users simply point their camera at anything that sparks curiosity and receive instant, contextual understanding without typing a single word.

One of our core features- AI Decode demonstrates this perceptual leap: when users upload an image, our multi-agent visual reasoning model doesn’t just identify what’s in the frame — it reverse-engineers the creative

and contextual DNA of that image, generating a precise, model-agnostic prompt that captures

its essence. This process, compact yet powerful, exemplifies how AI can move beyond responding to commands and instead become an intuitive collaborator that sees, interprets and acts, turning sight into insight seamlessly.

3.Why perception matters: richer collaboration, greater challenges

Richer collaboration: When AI understands more than prompts — interpreting context, modality (voice, image, gesture), State and intent- it becomes a collaborator rather than a mere executor. For example, students using generative AI to solve creative problems reported greater progress when they treated AI as a partner rather than a tool.

In human-AI research, Dalsgaard et al. argue that prompts are often treated as static interfaces, neglecting their role in mediating cognition and collaboration.  As AI grows perceptual capacities, image, video, audio, real-world sensors, the potential opens for human-AI partnerships that understand gesture, tone, visual context and emotion.

Greater challenges: At the same time, shifting toward perception brings new issues:

Trust & misalignment: If the AI “perceives” incorrectly, the human partner may be misled. The aforementioned bias-amplification feedback loop shows that perceptions matter not only for the AI, but also for human judgements.

Human psychological effects: When humans treat AI as social actors, attributing intention or “mind” to it, this can influence how they treat both machines and people. Guingrich et al. show that perception of AI as conscious may carry over into human-human relations.

Interface complexity: What used to be a typed prompt may become a multi-modal interaction- voice, gesture, environment. Designing for perception means designing for context, ambiguity, user State and emotion.

Ethical & privacy concerns: Perception often implies sensing- collecting more data about a user: their voice tone, gaze, environment, emotion. This raises serious questions about consent, bias, and autonomy.

4. Designing for perception: key dimensions

To move from prompt to perception, human-AI interaction must consider several interrelated dimensions:

• Modality & context: Traditional prompt systems focus on text. Perceptual systems incorporate vision, audio, environmental sensors, and user behavior. The richer the input, the more nuanced the response- but also the more potential for error.
• User intent & agency: Perceptual systems need to infer human intent, not just literal commands. This requires user modelling, context awareness, and explanation of system inferences. Wang et al. show prompt-supportive functions that help users articulate perception and refine prompts reduce cognitive load.
• Social cognition & perception: Humans apply social heuristics to machines. McKee’s work on warmth/competence highlights how perceptions of machine behavior affect cooperation. Designers must understand the social layer of interaction, not just functional correctness.
• Feedback loops: As stated above, AI perception has a bearing on user behaviors, which affect AI results. Systems must be designed for feedback loops to identify and reduce bias amplification, and we must be transparent about perception mechanisms.
• Ethical design and transparency: As mentioned before, increased perception brings with it greater ethical responsibilities in design. Whether or not the AI can truly be ascribed human-like capabilities, this perception may lead users to place excessive trust in it. Guingrich et al. discuss the ethical and behavioral consequences of this phenomenon.

Drawing from our experience as practitioners actively building in this space, we recognize that integrating perception into AI systems demands rigorous interdisciplinary collaboration and constant iteration. At Chance AI, we have designed our product with a deep commitment to contextual accuracy and ethical responsibility. Unlike generic AI tools that treat vision as secondary, Chance places perception at its core, ensuring that every interaction is meaningful, transparent, and free from commercial noise. This practitioner-informed approach bridges the gap between cutting-edge research and impactful product development, proving that perceptual AI, when built thoughtfully, can empower users to explore, learn, and create in ways that feel natural and deeply human.

5. Case study and future directions

Consider a generative AI for design: in the early prompt era the user writes: “Create a futuristic city skyline at sunset.” The AI responds with an image.

In a perceptual era, the system might observe the user’s real-world room lighting, posture, voice tone, ambient noise, and offer: “I can sense you’re sketching on-the-go and perhaps want a warm palette to match your current lighting, shall I generate a few variants in that aesthetic?”

This reflects a richer synergy of intuition and automation.

Future research is pointing toward models that merge multimodal perception, reasoning, and context-aware response. The “HumanSense” framework, for example, benchmarks models comprehending multimodal human interactions (text + audio + image) and responding empathetically.

Similarly, “Implicature in Interaction” (Hota & Jokinen) explores prompts embedded with meaning beyond the literal, contextual cues, shared background knowledge, tone, which can improve alignment and quality of human-AI communication.

6. Concluding thoughts

As human-AI interaction evolves from prompts to perception, collaboration deepens, becoming more fluid and human-centered. This shift is as much cognitive, psychological, and ethical as it is technical. System design must anticipate how perception transforms the user-machine relationship: from instruction-response to understanding-shared meaning.

Prompts remain useful, but perception adds the richness of context, intent, modality and relationship. The question becomes less “What can I ask the system to do?” and more “What does the system understand about where I am, what I intend, how I feel and how might that shape our interaction?” Efficiently navigating this shift will require interdisciplinary design: engineering, human-computer interaction, psychology and ethics.

For practitioners, the key is attending to human perceptions of AI: how social cues, trust, and bias evolve; and monitoring feedback loops linking machine perception, human behavior, and system output.

In essence, the next frontier in human-AI interaction is not just smarter machines responding to better prompts- it is machines that truly perceive our world and partner with us through context, intent, and meaning.

Author is a founder and CEO of Chance AI; views are personal