Go Module

Description

The Go module allows a 3rd party application to leverage Bakery functionality, without using a Bakery service:

Selectively remove streams from DASH and HLS manifests, via filters
Provide a API for CRUD operation of persistence objects:
- RuleSets
- Origins
- DubCreditRegistraions
- EvalScripts
- DeviceCapabilityRuleSets
Set client Device Capabilities
Set client location (equivalent to X-Geo-Asn request-header)

Note that a PostrgreSQL backend is required in order to function, although a Redis backend is optional.

Steps:

go mod init (for new projects, only)
go get -u -v github.com/cbsinteractive/bakery@feature/vendor_module
go get github.com/lib/pq@v1.10.5`
go mod tidy
go mod vendor

Usage

Import bakery module and required sub-modules

import (
	"github.com/cbsinteractive/bakery/common"
	"github.com/cbsinteractive/bakery/config"
	"github.com/cbsinteractive/bakery/filters"
	"github.com/cbsinteractive/bakery/repository"
	"github.com/cbsinteractive/bakery/repository/models"
)

Instatiate Bakery (see examples below)

DB configured via PG connection-string. No Redis. (Returns repository.Repository implementation)

var bkr Bakery
var repo repository.Repository
var err error
dbUrl := "postgresql://<dbuser>:<password>@<host>:<port>/<databasename>?sslmode=disable"

bkr, repo, err = NewBakery(PgConfig{DbURL: dbURL}, &RedisConfig{Omitted: true}, nil)

Bespoke `repository.Repository` implementation. `config.Config` object paritially overridden. No Redis.

var bkr Bakery
ver err error
conf := &config.Config{}

repo = func () repository.Repostiory {
   // construct custom implementation, here
}()
// override specific values in `conf` object, i.e.
conf.OriginHost = "https://nondefault.origin.host.com"

bkr, err = NewBakeryWithRepository(PgConfig{DbURL: dbURL}, repo, &RedisConfig{Omitted: true}, conf)

Bespoke `repository.Repository` implementation. No Redis. Default `config.Config` (via `nil` parameter)

var bkr Bakery
ver err error
conf := &config.Config{}

repo = func () repository.Repostiory {
   // construct custom implementation, here
}()
// override specific values in `conf` object, i.e.
conf.OriginHost = "https://nondefault.origin.host.com"

bkr, err = NewBakeryWithRepository(PgConfig{DbURL: dbURL}, repo, &RedisConfig{Omitted: true}, nil)

Default repository. Redis configured. Custom HTTP client

var bkr Bakery
ver err error
dbURL := <postgres connection>
customerHttpClient := <>

redisConfig := &bakery.RedisConfig {
   Host: "some.redishost.com",
   Port: "6379", // default Redis port
   Username: "", // set to empty-string, if not configured
   Password: "",// set to empty-string, if not configured
   DbID: 0, // default redis-database ID
}

bkr, err = NewBakery(PgConfig{DbURL: dbURL}, redisConfig, nil)

// override default HTTPClient
bkr.GetConfig().Client = customHttpClient

Filter

func FilterRequest(bkr *bakery.Bakery, request *http.Request) ([]byte, error) {
   // request path includes Bakery inline filters, i.e.
   // v(hvc)/l(fr)/path/to/resource/on/origin/manifest.mpd

   // apply inline-filters specified in URI, plus other RulesSet, EvalScripts, or DeviceCapsRuleSets
   // that are trigged by default, or by a combination of URI/path pattern, X-Device-Capabilities header, or X-Geo-Asn header
   return bkr.Filter(request)
}

Configuration

Bakery can either be configured via the config.Config object passed into the NewBakery method, or by setting default environment variables. Note that Bakery module requires a backing database – if none is configurated it will attempt to configure the DB-connection use default environment variables (see below)

A Redis backend is optional can be configured explicitly using the RedisConfig object, or via default environment-variables if the configuration is nil. To use Bakery module without Redis, supply a ConfigConfig which Omitted is true, i.e.

bkr, repo, err = NewBakery(PgConfig{DbURL: dbURL}, &RedisConfig{Omitted: true}, nil)

Environment variables

If Bakery is not configured explicity via the NewBakery constructor, it will attempt to configure itself via the below environment variable. Note that each environment-variable has a primary/preferred form (i.e. BAKERY_) and a “secondary” form (i.e. with the BAKERY_-prefix).
If Redis is configured, only the host:porl URL is required – user, pass, and database-ID are optional.

env var	primary	description
`BAKERY_POSTGRES_URL`	`✔`	postgres DB URL
`POSTGRES_URL`
`BAKERY_REDIS_URL`	`✔`	`host:port`-style Redis URL
`REDIS_URL`
`BAKERY_REDIS_USER`	`✔`	Redis user (optional)
`REDIS_USER`
`BAKERY_REDIS_PASS`	`✔`	Redis password (optional)
`REDIS_PASS`
`BAKERY_REDIS_DB_ID`	`✔`	integer Redis database-ID (optional)
`REDIS_DB_ID`

Filtering

Filters are prefixed to the manifest-path (i.e. specified inline).

Filter Types

constant name	string value	reference
`Videos`	`v`	videos
`Audios`	`a`	audios
`Captions`	`c`	captions
`Characteristics`	`chr`	characteristics
`ContentTypes`	`ct`	content-type
`Tags`	`tags`	tags
`Trim`	`t`	trim
`Bitrate`	`b`	bandwidth
`FrameRate`	`fps`	frame-rate
`DeWeave`	`dw`	de-weave
`PreventHTTPStatusError`	`phe`	prevent-http-error
`RuleSetFilter`	`rs`	rule-set
`OriginOption`	`o`	origin
`ReturnError`	`err`	error
`CreateHeader`	`ch`	create-header
`CreditStitch`	`cs`	credit-stitch
`DeviceCap`	`dc`	device-capabilities

See Bakery Filters documentation for further information on how to use filters.

Persistence Layer

The Bakery module also allows CRUD operations on persistence-layer objects:

Bakery module exposes a Get, Update, Create, and Delete methods for each of the objects above: i.e.

Get<ObjectName>s
Create<ObjectName>
Update<ObjectName>
Delete<ObjectName>ByID

Note: the EvalScript object has DeleteEvalScriptByName instead of a delete-by-id method.

Doppler Decisioning Engine Integration

The Doppler decisioning engine maintains a dynamic map of domain-to-optimal-CDN provider, on a per ASN basis, changes to which are pushed to DNS providers on a regular basis. However, since DNS propagation can be slow, the updated mapping also is made available via Redis cache by the Doppler Cache Writer service, which transforms a frequenly-updated list of Doppler CDN mappings (stored as a flat file of JSON objects, in cloud storage) into a Redis-cache representation of those mappings. The Redis-cache entries are then used by Bakery to perform direct domain-replacement, thus bypassing problems related to slow DNS propagation.

Flat File Layout

The flat-file contains an array of JSON object, i.e.:

[{
   "<From-Host Key>":{
      "ToHosts":{
         "<To-Host-Key-1":"<To-Host-Name-1>",
         …
         "<To-Host-Key-n>":"<To-Host-Name-n>"
      },
      "Decision":[["<Location-1>", "To-Host-Key-i"],...,["<Location-m>", "<To-Host-Key-j>"]]
   },
   …
},
...]

Simple example:

[{
   "87debgha6d41.airspace-cdn.cbsivideo.com":{
      "ToHosts":{
         "a":"87debgha6d41.airspace-a.cbsivideo.com",
         "f":"87debgha6d41.airspace-f.cbsivideo.com"
      },
      "Decision":[
         [
            "AU:NSW:4804",
            "a"
         ],
         [
            "US:OH:7018",
            "f"
         ]
      ]
   }
}]

Redis Representation

For each JSON object in the flat-file, Doppler Cache Writer creates <key>: <value> pairs in Redis of the form <from-host>___<ASN>: <to-host>. The above JSON object would produce following Redis key/value pairs:

87debgha6d41.airspace-cdn.cbsivideo.com___AU:NSW:4804: 87debgha6d41.airspace-a.cbsivideo.com
87debgha6d41.airspace-cdn.cbsivideo.com___US:OH:7018: 87debgha6d41.airspace-f.cbsivideo.com

Location Header

Bakery requests may contain an optional X-Geo-Asn header, with contains region + ASN information for the requesting client, i.e. the header/value X-Geo-Asn: US:MA:7922 identifes a request from Massachusetts, US by a client in ASN 7922

Origin Subsitution

Prior to requesting a manifest, the manifest-origin (i.e. mediag2481ed.airspacecdn.cbsivideo.com) is combined with the value of the X-Geo-Asn (if present), to form a Redis lookup key, i.e.:

mediag2481ed.airspacecdn.cbsivideo.com___US:MA:7922

If this key exists in Redis, then the value of that key (i.e. mediag2481ed.airspace-f.cbsivideo.com, in the above example) is substiuted for the origin-host, for for request.