WebAR architecture¶

Updated on April 15th 2021

Three layers¶

Argo webar is based on 3 main layers :

tracking engine
webArgo library, embedding a tracking engine
webapp, embedding a webArgo library

Tracking engine¶

The tracking engine is only in charge of tracking a marker in the camera feed.

webArgo library¶

The webArgo provides high level functionalities to webapp. In standard implementation it is based on Argo's APIs for :

marker recognition
3D interactive scene to be displayed in AR
data for statistics purposes

Webapp¶

The webapp is mobile website presenting the webAr to the end user. It is mainly a light website consuming the webArgo library. This library can possily be downloaded at runtime. For our own implementation, we decided to use VueJs.

About tracking engine¶

To be done

About webArgo library¶

The library is mainly based on component architecture and an orchestrator.

In a standard use case, you should only call the orchestrator’s apis.

In your webapp, you have to instantiate the orchestrator with some options of your choice.

Then you will need to start/stop it and listen to the events you need to display your UX.

Everything is displayed within a div whose id is “argo”. By default this div is fullscreen but it could be reduced/moved if needed.

NB : no UX is included within the library. It’s up to the library user to display the needed UX according to the events/statuses of the components.

Components¶

Each component has a name, a status and a list of error codes.

The current components are :

Video : in charge of managing the webcam with optimal size and displaying the feed
Tracking : in charge of finding and tracking a marker in the video feed
Scene : in charge of displaying the 3D overlay specified in Argo Editor
Reco : in charge of finding a marker in a database whether online or offline.
Odo : in charge of getting gyroscope data to better the experience. It is recommended not to enable it for now. If enabled, the engine will apply priority to gyro rotation over the visual information. It could be used in some rare cases with fixed images.

Status is a number from the following list :

LOADING,
READY,
STARTED,
RUNNING,
PAUSED,
SUCCEEDED,
FAILED,
COMPLETED

Code error examples :

general errors
- 0 : no internet or connection lost
video component errors
- 100 : video : needed api not available
- 101 : video : could not start video for unknown reason
scene component errors
- 200 : undefined loading assets error
- 200 : asset model not found on the server
- 201 : asset server error
- 202 : asset deserialization error
reco component errors
- 300 : undefined error for reco
tracking errors
- 400 : wasm not supported or such
odo component error
- 700 : gyroscope not available or not working
- 701 : DeviceOrientation not available

Each component has at least the following apis :

checkCompatibility : boolean
- returns true if the component is compatible with the smartphone and browser.
getStatus(): number
- corresponding to its status
getState(): json
- returns context information
canStart(): Boolean
- true if component can be started.
- For instance if Reco is being processing a demand to the online database, it will return false until the demand is complete.
start() :void
- Starts the component
stop() :void
- Stops the component
render():void
- called at every frame by the orchestrator

Orchestrator¶

Core apis¶

startCam() : void
- Starts the webcam
startReco() : void
- Starts the webcam and sends a marker recognition request to the server
startAR() : void
- Starts the webcam, sends a marker recognition request to the server and downloads/display the scene specified in Argo Editor
canStartAR() : boolean
- Returns true if all components are ready to startAR.
Stop() : void
- Stop every component, including the webcam

BehaviourManager¶

All interactions are managed through the Behaviour manager. A behaviour is the combination of a trigger event on an origin asset and an action. Every time a trigger event is sent by an asset (ex: click on 3D object), all behaviours based on this 3D object and trigger type are played.

Trigger events :

firstDisplay = 0 (triggered at first Display)
click =1 (triggered on click on an asset)
beforeDisplay =2 (triggered bdefore first Display of an asset. Can be used to hide an element at start)

Action types :

show =0
hide =1
playVideo =2
playAnim=3
togglePlay=4
fullscreen=5
openURL=6
openShare=7
click=8

A behaviour is described by a JSON like

{
    trigger : enum_trigger,
    origin : string,//name of the asset acted on
    target:string,//name of the asset we should apply something on
    action: {
      type : action type,
      params: an object
    }
}

Utilities¶

updateOptions(options) : will update options for all components
getReferences() : provides a list of useful links : 3D scene, lightState, etc
getFoundMarker() : provides all info you need on the tracked marker
getSnapshot() : image
- Returns a jpg image of the AR scene
toggleFullScreen(): void
- Enable/Disable fullscreen if possible in this browser
desactivateAudio/activateAudio() : void
- In iOS it is not possible to start sounds without a user interaction like a click.
- Like in youtube in its mobile browser version, you will need to add an “Reactivate sound” button. This button could call activateAudio which will “unmute” every loaded videos.

Options and explanations¶

Full example¶

let options = {
  app: {
    useGyroscop: false,
    maxGyro:1.0,
    autoStart:true,
    useShadow:false,
    minZIndex:1,//minIndex for webAR
    strategies:{
      "tracking":true,
      "afterLost":0
    },
  },
  apis: {
    dev_apiMediaUrl: "https://apidev.bear2b.com/",
    dev_apiUrl: "https://apidev.bear2b.com/v2.11/",
    dev_jwt: {token provided by Argo for a given webapp},
    prod_apiMediaUrl: "https://api.bear2b.com/",
    prod_apiUrl: "https://api.bear2b.com/v2.11/",
    prod_jwt: {token provided by Argo for a given webapp},
    default: "prod"
  },

  video: {
    videoId: "webcam",
    trackConstraints: {
      facingMode: {
        ideal: "environment"
      },
      width: {
        min: 480,
        ideal: 640,
        max: 1000
      },
      height: {
        min: 480,
        ideal: 480,
        max: 1000
      },
    }
  },
  reco: {
    pageId:"",//set as 717039 for example if you want to skip recognition step and apply AR to a given maker (ie pageId)
    maxRecoRequests: 1,
  },
  tracking: {
    useWorkers: false,
    nbWorkers: 1,//nb of workers. 1 is recommended.
    maxNbNotFound: 5, //if the webapp did not find the markers maxNbNotFound times in a row, assets will be hidden or shown in VR,
  },
  scene: {
    showMarker:true,
    three: {
      motherDomElement: null,
      antialias: true,
      preserveDrawingBuffer: true,
      alpha:true,
      focalLength: 424,
      canvasId: "myARCanvas",
      ftsRotation:{
        x:-0.5,y:0,z:0
      }
  }
}

Details about main parameters¶

app.autoStart¶

If autoStart is set as true, the library will try to start webcam immediately when loaded. Otherwise it will wait for startCam().

reco.pid¶

pid is the unique identified of a marker. If it specified, the webArgo library will skip the recognition step and directly consider that this marker should be tracked. Example : in https://editor.ar-go.co/index.html?bookId=56204&pageId=787766, the pid is 787766

This is very nice in case your webapp is one for one marker only !

tracking.useWorkers¶

If useWorkers is set as true, the library will use a 'worker'. Using workers will use multithreading capacities of the browser. Overall experience should be smoother but tracking may look more floating.

tracking.strategies¶

If strategies.tracking is true, the tracking engine will be used; otherwise, the webapp will not try to track the marker.

strategies.afterLost indicates what should be the behaviour is the tracking is lost :

0 : fly to screen
1 : fadeout, the scene will fade out if tracking is lost
2 : gyro only. This is an experimental feature, not to be used in production for now. It should be coupled with strategies.tracking = true

scene.showMarker¶

If showMarker is true, the webArgo library will show the marker image in fly to screen mode.

scene.three.ftsRotation¶

Once in fly to screen mode, we can specify default rotation angles (in radians) :

x : rotation around the horizontal axis
y : rotation around the vertical axis

Basic use in a webapp¶

Add argo.js script
Create an instance of the orchestrator :
1. const webArgo=new ARGO.webArgo(options,onWebAppReady);
2. Options being a json including many optimal parameters such as
  1. Specify a default marker to bypass the marker recognition on the server
  2. Should the webcam start automatically
  3. Tracking options : could be changed if you’re sure your marker is fixed
  4. Use workers
3. onWebAppready to be called once the library is ready to run
Check the compatibility of the browser :
1. webArgo.checkCompatibility()
Listen to the events you need
1. Example : webArgo.subscribeTo(1, [1, 2, 3], onVideoStatus); will call onVideoStatus every time the video component will change status to 1,2 or 3.
Catch errors thrown by the library.
1. window.addEventListener("argo_error", function(e) { onError(e); }, false );

Other events¶

Every time a 3D object is loaded an “argo_loaded” event with object name in details field is sent.

Every time a 3D object is clicked an “argo_click” event with object name in details field is sent.

NB : to me updated

Light management¶

It is now possible to modify lighting parameters. Here is how :

get the lightState : webArgo.getReferences().lightState Modify whatever you need.
Apply your modifications : webArgo.getReferences().lightState.update()

NB : not possible to add/delete lights in current version

Advanced tips¶

Advanced interactions in Argo editor¶

In some cases, you wish you could add highly customized interactions. Of course you can do it on your side. But you can also, use the Argo editor :

Open the editor, and add a webview asset in your scene
Name it 'interaction' (only the beginning is considered, so interaction_trex is fine)
Code in javascript in it. This code will be invoked when the scene is loaded

We will provide some examples of code. But here are the entry points :

start with webArgo.getReferences()
code based on threejs. You can get reference to threejs, scene, camera, renderer, etc from getReferences()

Example 1: using behaviours¶

Here is an example to hide image 2 at start and show it when you click on image1

const references = webArgo.getReferences();//we get access to some webArgo internal controlers
const behaviourManager = references.behaviourManager; //behaviourManager is the controler for every trigger/action

//if I click image1, I show Image2
let origin="image1"
let target="image2"
let behaviourShowOnClick = {
    trigger: 1,
    origin : origin,
    target:target,
    action: {
        type:0
    }
}
behaviourManager.addBehaviour(behaviourShowOnClick)

//When it is first displayed, we hide image2
let behaviourHideAtStart = {
    trigger: 0,
    origin : "image2",//we could put anything here in fact
    target: "image2",
    action: {
        type:1
    }
}
behaviourManager.addBehaviour(behaviourHideAtStart)

About Webapp¶

How it works and why¶

If you use our default webapp: https://web.ar-go.co You start with our landing page.

Click on the 'Launch' button to start the camera. A semi-transparent rectangle intices users where to get the image they want to recognize.
Once you're ready to test an image, click on the scan round button. The webapp tries to recognize the marker you are aiming your phone at. This recognition step will look for candidates in Argo platform.
If a marker in the database has been recognized, the webapp will download the 3D interactive scene and show it in augmented reality on the found marker. If not, webapp shows you it failed to recognize your image. Note that now you see a Camera button, if you click on it, it will take a snapshot of your screen and display it above the 3D scene. Press it long to share it or save it.
If the marker is lost (tracking failed), then 3D interactive scene is displayed in 'Fly to screen mode', meaning you can rotate it and zoom/unzoom with fingers.

Customize with parameters¶

You can add parameters in your url to get some custom settings. Examples :

https://web.ar-go.co?as=1, the webcam will start automatically.
https://web.ar-go.co?pid=xx, the marker is known, thus the recognition step will be skipped. This is a very good idea when you know the marker in advance. By default, when the webcam starts, you will see the specified marker in semi transparency, helping users know what image they need to look for.

And of course, they can be combined.

List of current parameters :

as =1 will autostart webcam
pid= xx will skip recognition and assume marker id is xx
us=1 will try and use to apply shadows on 3D models and 3D models only !
client_id= xx will use db of this client
prod=0 will use dev database
s_t=0 (tracking strategy)webapp will not track marker after reco. It will play after tracking lost strategy
s_al=0 or 1 (after tracking lost strategy) 0 : will display fly to screen if tracking lost. 1: will fade out if tracking lost. 2 : only gyroscope display. 3: STQR like display, you decide where to put the scene and then you use gyroscope.
sm =0 or 1. If 1, marker will be displayed in fly to screen mode.
fts_x = angle in radian => for default angle view in fly to screen mode. 0 is from top. Default is 0.5

Customized webapps¶

For premium customers, it is also possible in the Argo manager to :

completely change the landing page
set up some above parameters
have a customized url, like yourchoice.webargo.app

Manage translations and languages¶

You can change translations and add languages by using javascript; for instance when personnalizing the landing page.

You can add a language by using international codes.

Language will be chosen by the browser language.

You can also use your own language to force a translation.

Below is a simple example to change the text displayed when you scan

let messages=app.i18n// all the translations
messages.fr.placeholderText="Scannez une page"
messages.en.placeholderText="Scan a page"
app.i18n.setLocaleMessage("fr",messages.fr)
app.i18n.setLocaleMessage("en",messages.en)

For advanced usages, please refer to https://kazupon.github.io/vue-i18n

Change options (experimental)¶

You can access and change some deep options of the webapp. Be careful, this is an experimental feature.

Below is a simple example :

let options=app.options// get the options
options.reco.pageId=657777//we change something
if(webArgo != undefined){
    webArgo.updateOptions(options)//we apply the options
}

Change icons¶

Icons from the webapp List of icons : - toFullScreenIcon - toNormalScreenIcon - iconScan - iconScreenshot - iconDownload - iconBack - iconClose - iconCloseCircle - iconAudioOff - iconAudioOn

Add this in landing page script. Of course, as many as you want of ocurse

  window.appCustom = {
  webcam: {
    iconScreenshot: 'https://api.bear2b.com/media/208259507661bc9cf03f5db2.34113105.png'
  }
}