The player's semantic experience

In my first posting I outlined what it means for a player to communicate at will without distance or devices. Just as we have conversations with people in the room we are in and our friends hear our talking and respond, that in the future we can have the same natural interaction when we want to accomplish anything digital. The excitement of this statement is when we interpret “anything” as a “rich communications experience”. We are no longer restricted to verbal communications between two people. Secure natural communications includes all communications and computing interactions that exist for people and devices - for human communications as well as device communications.

I also introduced the five key constructs required to achieve this new capability and infrastructure. We require support for the player’s semantic experience, the player’s physical experience, the decode engine, core infrastructure, and information intelligence. If you have been following some of the dialog on this blog you know that Chiku has proposed that we should also focus on new enabled applications as a sixth construct.

In the next set of postings I will elaborate on each of these constructs and also discuss some of the research problems that can be addressed to further improve capabilities

The player’s semantic experience

To better characterize how the player will intellectually interact with other players, we characterize the following components of the player’s semantic experience. In the infrastructure we build for secure natural communications we will require support for each of these components.

• The players. The players in secure natural communications are the potential parties to communications. In today’s terms it would be anything with an IP address. This includes people, computers, devices, sensors, microphones, cameras, items with RFID tags, MP3 players, anything with a GPS system attached, scanners, medical imaging devices, etc. So the notion of “natural” communications means that each of these devices is able to send and receive communications in a way that is natural for them. And any user is able to naturally refer to these items and find them.

• The groups. Group communication has become a fundamental paradigm through the advent of social networking. The infrastructure must make it natural for users to participate in arbitrary ad hoc groups.

• Media. Secure natural communications must support unstructured data as well as voice, video, high quality music, fax, etc.

• Semantics. We have developed increasingly sophisticated means for people, computer systems, and applications to communicate with each other in a highly coded fashion. To the extent that the current protocols and interfaces are adequate, they would be used as they are defined today; but many need to be made more natural.

With these four components, we can give the next level of description of secure natural communications.

Since we include all players imaginable this clarifies the form of communications that is supported. Any player can communicate with any other player. Each has a set of requests supported and each has a different sense of what is “natural”. The infrastructure supports both their physical requests and semantic meeting thereof.

“Groups” recognizes that much communication wants to be multi-party. Increasingly, large projects cross organizational boundaries and collaborate on some set of tasks. Hence all types of group requests must be intelligently anticipated by the infrastructure. When a person or device is participating in multiple group interactions, they must be able to handle the challenge to participate in multiple groups. To achieve this, the infrastructure must present all of the information from multiple interactions in the most intuitive possible fashion. They must handle multiple group interactions digitally as well as any human can handle multiple physical interactions. The infrastructure needs to support an exponential number of potential groups that want to form.

“Media” requires that we digitize everything, so we are encompassing all of human experience in this infrastructure.

The Semantic layer implies that compute tasks that happen with a regular pattern each develop their own means of semantic expression which is also supported.

Of course, in describing each of these components of the player’s semantic experience, I am only scratching the surface. As we develop this, each of these will get more detail. The readers of the blog are invited to provide their viewpoints by commenting on what they think some of these details should be.

In the next posting, I will describe efforts that should take place to more fully develop a player’s semantic experience.

Secure natural communications - achievability and relationship with computing

Today I’ll address two questions that continue to come up.

First: with secure natural communications people can walk around the streets not holding any computing device and look up some piece of data or request a computation from the computing cloud, and it gets performed by an intelligent infrastructure. I’m asked - how realistic is this? People struggle to understand the vastness of the infrastructure needs to achieve this lofty goal. Are parts of this “vision” beyond our grasp?

Here is how I think about it. There are more and less demanding variations on secure natural communications. Enough technology exists that a proof-of-concept could be built, today. Over a long period of time technology and infrastructure advances will make more of this possible. Key to making this happen, however, is to set a stretch goal so we can plan the required physical infrastructure and motivate the necessary innovation.

Second question: Information infrastructure, as in today’s wireless infrastructure, Internet or World Wide Web provides more than communications. Most of the value is in the applications - e-commerce, auctions, advertising, document interchange, compute intensive applications, storage. How does our focus on communications relate to the larger computing applications.

Here is how I think about this issue. At the moment, this blog is focusing on the core infrastructure rather than the compute tasks because for each of these tasks, there is a dominant common communications infrastructure need. To be sure, communications takes place at every level: from physical transmission of bits at the lowest level to the interpretation thereof by applications that have application level protocols.

In particular, when we discuss “the player’s semantic experience”, we will ultimately support the semantics for every type of computing task. However, with the existence of millions of applications this blog will not address application specific protocols herein – although some require substantial infrastructure of their own. That work must proceed in parallel with the core communications infrastructure.

Next blog posting I will next move to a detailed description of the five constructs mentioned in the introductory posting.

Secure natural commmunications - elaboration

Until now, my description of secure natural communications has been conceptual, with a few examples. Before we go further it is useful to tease out more of what the words mean in the original statement describing secure natural communications:

Any player (person or device) can have a rich communications experience at will with any target without encumbrances of distance or additional devices.

Here are some elaborations of the above goal.

A rich communications experience. Included in this would be:
• Finding someone I’ve never communicated with
• Having a record of the relevant threads of information that I previously have with this group.
• Having the universe of social networking capabilities at my fingertips.
• In particular, sharing all types of data, using all types of media
• Being able to tie in the history of communications that are relevant.
• A range: from purely communicating to requesting a complex computing task with sophisticated semantics
• Security and privacy

…at will. Included in this would be:
• I don’t need to plan to find addressing conventions to locate the other player.
• I can use my natural means of communicating and a sensory rich world will understand my physical gestures and understand the semantics
• Speech recognition, gestures, natural language, are supported
• All media types are supported
• There are means (such as pervasive video screens or wearables) for me to get responses

…with any target. Included in this are groups, so it would include:
• Being able to create new groups with security and privacy
• Being able to create ad hoc groups to achieve some new task without a great deal of overhead – and still have the rich communications experience.
• Being able to join groups and get access to their shared history

…without encumbrances of distance. Included in this is:
• Assuming the very high bandwidth backbone in which bandwidth is not a constraint.
• Assuming the very high bandwidth local wireless network that gives adequate untethered bandwidth to get to the high bandwidth fiber backbone
• Developing protocols that work well with this unlimited bandwidth
• Recognizing that for certain purposes there are still constraints of propagation delay

…or additional devices• Implies that there are natural user interfaces (speech, gesture) that can be relied on; i.e. the technology is good enough
• Implies that when using such natural interfaces in a noisy area that my specific sentences and commands will be heard
• Understands that this might require nano-scale wearable microphones
• Any language
• Recognizes that many people will want to continue to use keyboard for input
• Recognizes that a variety of display and storage devices will continue to be needed

This provides a more complete description by example of the type of capability that is desired. Next posting I’ll provide a little more general background, before moving to a more detailed discussion of the constructs that can deliver this and the technologies that are needed.

Secure natural communications - how to proceed

After my initial posting last week, the most frequent question that I received was: how do I plan to proceed with this project? Is this something that we will build tomorrow? Is there some spec? Are these some wandering ideas?

Most of these questions came through email from my colleagues. This blog should be an active discussion community so please post your questions and comments to the blog – as several people have already done.

Reprise – and what is possible today

To reprise, the world should provide an infrastructure where: people can have a rich communications experience at will.

From a core technology perspective, almost all technologies required exist today in elemental form. There are numerous sensors that exist: often called by names such as microphones, cameras, displays, etc. These sensors can detect speech, movement, typing or provide images and videos. Properly configured, they provide the elements of natural human communications overcoming individual device conventions. With ample communications bandwidth we also overcome distance constraints. An infrastructure could both sense bits and interpret them – although not with perfection.

From this - in principle – one can take people’s utterances and build a management system that acts upon them. These can be primitive utterances that request basic communications. Or coded machine commands that request the performance of sophisticated compute function, rich applications, moving function to the cloud, etc.

What is not possible today

There is no infrastructure that provides secure natural communications. Not across the globe; not in a country; not in a city – not even on a campus. It would be considerably expensive to build and there is no known economic model to pay for this new infrastructure.

Goal and method

Here is my pitch. If it is possible to create such an infrastructure – and it would mimic how people naturally communicate – isn’t that compelling enough?

Hence my quest. I want to use a series of postings to express my views about the components of such an infrastructure. I need experts to add precision to this with their knowledge and views. After a while, we will develop enough clarity of viewpoint that we can prototype it on a small campus. If people like it – we will grow it from there.

Next several postings

So my first few postings will continue to discuss the high level concept of secure natural communications. I want to engage in discussion on the general idea. After that, I will elaborate in detail on the five constructs that I outlined last week. Provide a more complete description. What technologies exist? What more do we need to develop over time? And in discussion I will continue to need experts who have the domain knowledge to add precision.