ColorizerJS — My First Hack Project at eBay


First blog on the eBay Tech Blog, whooooh!

Hello! This blog will mostly be about my project, ColorizerJS, but since this is my first post on the eBay Tech Blog, I thought I’d introduce myself. For those unaware of some of the terminology and nomenclature, I will try to provide some brief short-bracket explanations[“[]”]. If you are submerged in web software or already cool enough to know the terminology, just pretend like there are no brackets.

I started working at eBay in May after graduating from the University of California, Santa Cruz, with a bachelor’s degree in Computer Science Game Design (now called CGPM). CSGD at UCSC was a great program for front-facing developers, as games are as much about the user (or player) as they are the algorithm. Plus, at least at UCSC, I got a whole lot more experience with managing assets and interfacing between technologies in my Game Design classes than in my vanilla CS classes.

The interview process at eBay was great, not too niche-specific and not too broad, with great questions on both niche and broad ends of the interview spectrum. I was hired as a full-time IC [individual contributor] on the Search Front-End [abbreviated to FE, meaning closest to the client] engineering team. This means that whenever you enter a search query, the stuff you get back — JavaScript [functionality, the bulk of it], CSS [look and feel], HTML [structure] — is what my team handles.

I was lucky enough to be on a great team at a great time. When Hack Week 2016 rolled in mid-June, I was too new to have many responsibilities keeping me from joining the Hack Week, yet experienced enough to know our codebase a bit and have some ideas for my potential project.

After some thought, I determined something that would be fun to work on, enhance the user experience, and bring in revenue to eBay through increased buyer activity. The initial problem arose from eBay sellers often having non-standard sized images, either shorter or thinner than a square. With the current redesign of Search Results Page (SRP), the method to combat this problem was to overlay all images onto a grey bounding square. This technically fixed the problem of non-standard image dimensions, but it did not lend itself to good aesthetics. This is where ColorizerJS comes in.

Current background implementation in production — not so hot

I was inspired to do ColorizerJS by my coworker (and one of my many mentors) Yoni Medoff, who made a script that drew loaded images to an HTML Canvas, averaged the RGB [red, green, and blue channels. You definitely know this. These brackets are so condescending.] values of each image in the search results for a query, then created an RGB color out of those average channel numbers, and set the containing square (normally just white or grey) behind the image to be the average color. This was intended to provide a more visually appealing way of standardizing the size and shape of each image in search results than just slapping them on top of a grey square. Yoni’s implementation looked significantly better than just a grey square, but it could definitely be improved upon.

There were four problems with his approach:

  • It did not look good in many cases. Sometimes the generated average does match any color in an image. A strong foreground color can throw that average out of whack, and most of all, the average may not resemble the “background” at all.
  • It depended on HTML <img/> elements [images on the (meta)physical webpage], which limited use to the DOM, so that isomorphic [using the same code in a browser and on the server] server-side use was impossible and parallelized web-workers [multiple CPU threads running JavaScript] wouldn’t work.
  • It converted the <img/> to an HTML Canvas element to extract the pixel data, a slow process that required drawing to large Canvas objects before being of any use. This was not time- or memory-efficient.
  • It was very tightly coupled to our codebase, meaning that it could not be used in other parts of the site or on other websites without huge changes.

My code running average algorithm — it’s alright

I thought, “Oh snap, here’s my chance to win $10k and become VP of engineering or whatever the Hack Week prize is.” Easy! All I needed to do was come up with a way to load the image into the browser cache to reduce double-requests [when a browser makes two requests to the server for the same image, BAD] and then read that cached image as binary [1000100101], analyze the binary file to extract its RGB pixel data, organize that in a logical way to analyze, and then analyze it and determine the most likely background color of the image to be appended to the background bounding square of the image, all while keeping the analysis code modular so as to be used on the client, web-worker, or server, and independent of our codebase and modular so that I could open-source it as an NPM module to be used by anyone on any page. Mayyybe not so easy.

So now I have a considerable job — meet all these criteria in a week — but where do I start? I decided starting with a client-side [user’s browser] image decoding [decoders analyze an image and give back a string of numbers indicating the channel values of each pixel] NPM module. eBay supports JPEG and WebP (Google’s new super-great image compression format) formats, and I figured since JPEG was older, I’d have more luck with it, so off I went looking for a client-side decoder. There weren’t any client-side JPEG decoders. Only Node [server-side]. Nice. After a few PRs [Pull Request] to jpeg-js (to support native UInt8Arrays instead of just Node-specific Buffer arrays) and just like that I had a client-side JPEG decoder. Nice.

Next I had to figure out how to request the image file as binary, and found a great article on JQuery Ajax custom transport overriding. This allowed me to send the data to jpeg-js as a bitstream in a JavaScript typed-array (UInt8Array in this case). jpeg-js sadly only supported 4-channel (including alpha-transparency) output, so in my color analysis code I handled both 3-channel and 4-channel output flags as 4-channel. This increased data overhead by about 1/3 more bytes per image [since each channel of a pixel is one byte] — inconvenient but not a dealbreaker.

With my now merged pull request to support client side analysis and array type, jpeg-js analyzed my binary input (after some array conversion) and gave me back an object with a height, width, and array of bytes [number from 0 – 255], each four array indices corresponding to a pixel in the image binary. I found a great WebP analysis library called libwebp (I got the code by viewing the page source) and got it working.

Now it’s time to do some background analysis!

I started with the simple average-of-the-edge-pixels algorithm and appended the resulting color to the bounding box behind each image result, which expectedly yielded sub-par results, but at least the analysis was working. I then I decided to up the size of the pixels analyzed to around 10 pixels per side. If the image was tall I would check the outer 8 pixels on the left and right side, if it was short I would check the top and bottom. I made a function that determined two modes for each color channel, determined the weight of each mode against each other and against the average, shaving off outliers and conversely, fully using the mode if it was significant enough in the image. This yielded great results, especially for images with fairly flat background colors or heavily dominating colors.

Lookin’ pretty good

But some images, especially those with sporadic colors, colors that are very different from side to side, edges with very different colors than the rest of the image, borders, and other complex images, sometimes did not cooperate with this implementation. Some generated colors would clearly not occur at all in the picture or just not fit as a background.

WHERE IS THAT MINT COMING FROM? — Mr Nice Guy could have fixed this if he were still here.

I did not want this algorithm to be just KINDA similar to the background for an image, I wanted it to be almost EXACT. I came up with a few modifications, more modes, more pixels analyzed, some analysis of sectors and their potential significance to determine their weight, and doing analysis of 2 pixels as well as large sections of 20–30 pixels from each edge and determining their weights. Also I fine-tuned the cutoffs for modes and averages to be more likely to exclude an average and include a mode. Some modifications to weights for each sector was required before I came up with a fairly finished product.

Niiiiiiice. It actually looks like a background.

Particularly impressed with the fighter jet analysis, good job Colorizer

I presented the Colorizer at the end of the week of Hack Week 2016 and got some positive responses from the judges. Fun project, and some hard work for a little more than a week.

I hope to make all magic numbers into parameters that can be defined by the user to widen the range of use cases that ColorizerJS can apply to. I also want to make some more functions in the Colorizer that can make it work with path image files to be used on the server, satisfying my original requirement of the module being isomorphic. If it ends up looking good, I might also come up with separate background colors for each side (top and bottom or left and right) if they are different enough. This would especially work well with unprofessionally shot images in sellers’ homes that might have vastly different upper and lower backgrounds in their images.

Soon I will get the code up and tested on Search Results Page, working with Web workers, and published as an NPM module with some examples of usage so that you and all of your friends can do some sweet image analysis. All that is needed is an image URL passed to ColorizerJS, the image is cached and analyzed, and out comes the background color!

Thanks for reading. I will update soon with more info as the project progresses. When I open-source ColorizerJS, I will post a link to the Git repo.

Big thanks to Yoni Medoff for the idea and initial implementation along with encouragement along the way, Eugene Ware for jpeg-js and the help getting it implemented, and Senthil Padmanabhan for the inspiration to write this blog.


Types of Content at eBay: Titles


All eBay-generated content is currently translated by our talented localization team, whereas eBay’s user-generated (UG) content is handled by our Machine Translation (MT) engine. It is common knowledge that UG text can get pretty noisy due to typos, non-dictionary terms, etc. At eBay, however, MT deals with more than that. We work with multiple types of UG content — search queries, item titles, and item descriptions — and each presents its own challenges. In the previous post we talked about search queries. This post discusses item titles.

Item Titles

Translating item titles (IT) provides our buyers from Russia, Brazil, Spanish Latin America, France, Italy, Germany, and Spain with an option to view eBay search results in their own language. This allows customers to look through pages of results and make an informed decision on which listings to open, because an image alone does not contain enough information. Being able to read and understand item titles is essential to a positive customer experience, which is why we invest a lot of effort into improving the MT engine for titles.

This type of content is very specific and presents a number of challenges for MT.


A title is a summarized item description composed of keywords. The eBay Help article on writing effective titles encourages sellers to omit punctuation and avoid trying to create a grammatically correct sentence. Following these and other tips is supposed to help sellers create a clear picture of an item and a good first impression, so it is important that the MT translation meets the same expectations. However, the lack of syntax and punctuation presents a problem for an MT engine that is normally trained on sentences. If it tries to translate a sequence of nouns, adjectives, and numbers as a sentence, meaning errors are unavoidable. It may start looking for a subject and a predicate and in general for a sentence structure, thus translating adjectives as verbs, moving words around, and so on.

As an example, let’s take a title for a can of paint: “20g Glow in the Dark Acrylic Luminous Paint Bright Pigment Graffiti Party DIY”.


What might go wrong here?

“Glow” may get translated as an imperative form of the verb, and “dark acrylic” — as a noun phrase with “acrylic” being a noun. (as in “Stay in the shaded area!”) – and that is just part of the title. Similar transformation may happen with polysemous words or those that belong to different parts of speech: “can”, “paint”, “party”, etc. The result of such translation may be a completely different item.


This is closely related to the previous issue. Segmenting a title and correctly identifying semantic units is of utmost importance for machine translation. For example, “Gucci fake snake leather purse”: in case of an incorrect segmentation, we may get a translation of a “Gucci fake” instead of the intended “fake snake leather”. Such translations are the most dangerous because they sound correct and believable yet present misleading information, which in the end may leave both a buyer and a seller unhappy with the experience.

To address these major issues, the science team created an engine just for item titles; it is trained on separate data sets. In addition, they have been working on a named entity recognition (NER) algorithm that identifies semantic units in a title before it goes in the MT engine for translation.


Sellers tend to use multiple synonyms in a title assuming this will increase the chances of matching search queries and coming up high in search results (which is a common misconception). For MT this means several things:

A chain of adjacent nouns or adjectives that are in no relation to each other

The machine needs to learn to translate them independently of each other. This is similar to the first issue described above, because the engine may try to create agreement where there should be none.


Example, Baby Toddler Kids Child Mini Cartoon Animal Backpack Schoolbag Shoulder Bag

We see four synonyms for the age reference and three synonyms for the item itself. The age reference terms are not all adjectives nor can all of them be translated as adjectives. Even a human translator would have to get creative and produce something like “for a baby/toddler, kids’, child’s” – because we could not simply leave all four of them as nouns; it would sound too abrupt and possibly confusing. The task is much more challenging for a machine. Not only should it avoid creating noun phrases (Kids child may turn into a kid’s child), it also needs to rephrase or insert prepositions where necessary (baby toddler child -> for baby, toddler, child; kids –> kids’). The best ways to approach this would vary depending on the target language.

Agreement with the head noun

In our example, there are three synonyms for a head noun: Backpack – Schoolbag – Shoulder Bag. What if they are of different gender in the target language? Which one should the adjectives agree with? A human translator would probably pick the first one, but MT may not think the same way. Here is a bigger challenge: the head noun does not immediately follow the adjectives describing it. In our example there are two other nouns between the attributes “Kids Child” and the head noun “Backpack”. The machine is supposed to figure out that “kids” describes “backpack”, not “cartoon” or “animal”. As you can imagine, however, the most logical decision for a machine would be to connect “kids” with “cartoon”.

Agreement plays a very important role in translating item titles, because it provides a customer with a description of features and qualities of the item. If you connect an attribute with the wrong noun, it will modify an incorrect object and produce an overall misleading translation. In our example, with the incorrect agreement, a user will read: “backpack with a kids’ cartoon animal”, which is in essence a different item than a “kids’ backpack with a cartoon animal”. One may argue that an image would be a clear indication that the item is a kids’ backpack. Unfortunately, a picture is not always a reliable source of information. In our case, there are similar backpacks for adults, which is why an accurate translation will make a difference.



Sellers use multiple acronyms to save space and fit as much information in a title as possible. For MT this presents several challenges.

  • Rare, unknown acronyms or acronyms that sellers made up on the spot. Gathering more training data and compiling additional lists of expanded out-of-vocabulary (OOV) acronyms is helping address that.
  • Polysemic acronyms that have different translations in different categories. The most challenging acronyms are the ones that have more than one meaning in the same category. For example, “RN” appears in Clothing, Shoes and Accessories as “registered nurse”, “Rapa Nui”, “Rusty Neal”, and as part of model names for Nike, Hugo Boss, A&F and other brands.

Writings and names of songs/music bands/movies/video games

This is common content for certain categories. Singling out a movie or song title out of the rest of the string may be difficult because there is often no contextual information pointing to the fact that it is a movie or a song. It is not much of a problem in the DVD or Music category, but quite often you will find reference to a movie title or a music band name in other categories such as Collectables or Clothing. It is also common for sellers to quote a writing on the item they are selling. Ideally, we would want to have the writing to be left as is so that the customer would know exactly what the item depicts. As you can imagine, however, literally anything can be written on a t-shirt or a poster, which is why it is very difficult for a machine to differentiate a writing from the actual item description. In such cases a user would have to rely on the quality and size of an image, which may not be the best on the search results page.


In this example, “New York Vermont Quebec” is part of the poster design, but it is barely visible. In the text of the item title, however, it may be interpreted as locations of the poster, places it originally came from, etc. Identifying this as verbatim writing, thus keeping it in English, would be a very difficult task for an MT engine, but it would clearly benefit an eBay customer.


With so many aspects to keep in mind, training the engine to translate eBay item titles is certainly a challenge. Our teams of scientists and linguists are actively and successfully working on ways to improve the quality of the training data and the MT output.

The API Journey: Or How We Built a New Family of Modern Services


An API — or application programming interface — is an intermediary that enables applications to interact. It is a contract that specifies how applications talk to one another. Further, an API creates a separation between a service provider and its consumers. Essentially, it decouples their implementations. As long as the contract stays intact, API providers may continue changing their code and underlying dependencies without destabilizing clients.

APIs are a big deal. After dealing with SOAP-based legacy APIs for years, eBay started a journey to deliver a new, modern family of APIs for sellers and buyers. Our principal goal was to design a set of interfaces that will meet business objectives, attract developers, replace our legacy APIs, and be long-lived. This is not an easy job. As I mentioned, APIs are a contract, and as such, they cannot be changed in ways that break existing integrations. APIs should evolve and grow with the business, so they must also be expandable and flexible. Now, that is hard.

Our challenge was to create a vision for the API, plan ahead and design a stable contract that will last for years, even as we add business capabilities. Here is how we did it.


“Ensuring and validating that assets and artifacts within the architecture are acting as expected and maintaining a certain level of quality.” — Gartner

To achieve consistency across the APIs, we followed a governance process and compliance model. One of our most important goals was improving the quality of the APIs by defining and enforcing standards and policies. We established a governance process that was objective, transparent, manageable, and explicit. Our compliance model for web services is platform- and tenant-agnostic and fits well into eBay’s overall API maturity model. Levels of compliance are specified by a set of characteristics and capabilities that may be measured and assessed. This helps to identify and quantify a realistic target maturity for our APIs in a given period. (And, it is testable!)

Design and beyond

“Unfortunately, people are fairly good at short-term design, and usually awful at long-term design.” — Roy Fielding

First and foremost, the API blueprint is the starting point. At eBay, a blueprint describes detailed API design enough to verify, implement, maintain, and extend capabilities in the future. Designing APIs has many analogies to building a house. Without a proper blueprint, pressure to deliver on time often leads to a poor design. To further frame an analogy to house construction, working without a blueprint causes shortcuts similar to building a bathroom in the kitchen since the plumbing work has already been done there. The challenge lies in finding a balance between our agile product development methodology and time needed to come up with a detailed design. Implementation becomes straightforward once there is a blueprint and clear understanding of what needs to be done.

For our new family of APIs, we followed our interface design method (“IDM”). The IDM is the process of arriving at an underlying model and a stable contract. It starts with capturing use cases by specifying actors, concepts, and actions, and then deriving entity relationships. Further, nouns are identified from the entities and verbs from the actions. The final phase of the IDM process is determining resource representation and specifying authorization details.

Use cases

Actor Use case Description Constraints
Seller Seller creates an advertising campaign The seller creates an advertising campaign and specifies a name, effective dates, funding strategy, and a criterion that defines inventory partitioning. The advertising campaign is either a rule-based campaign, where listings are auto-selected according to specified inventory partitioning criteria or a campaign with listings added by reference. A listing belongs to only one effective campaign.

The currently supported funding model is the cost per sale (CPS).

Inventory is partitioned based on following dimensions:

  • eBay-defined item categories
  • Seller-defined item categories
  • Item price range
  • Brand
  • Fulfillment cost
  • Item condition
. . .

Entity relationships diagram



HTTP request Authorization (OAuth 2.0 scope)
POST /sell/marketing/v1/ad_campaign
. . .

We followed pragmatic RESTful principles with standard implementations around cross-cutting concerns: error handling, security, behavioral tracking, and operational metrics. APIs represent the consumer’s view of the capabilities, and the URI names must be meaningful to developers. Our URI pattern takes a consumer-centric approach by providing a consistent, predictable, and understandable names across APIs. This pattern makes an API intuitive and easy to discover and consume. In most of the cases, the new APIs use JSON for resource representations. It is compact and easy to parse and translate. For certain use cases, supporting additional formats is straightforward, since our RESTful architecture style leaves room for such flexibility. So far, we have managed to stick to standard formats and media types. OAuth 2.0 protocol is leveraged to address security and data privacy concerns. Here, the challenge was to balance the need of fine-grained scopes that protect data and activities while managing the scope policies.

APIs are more than pure design and implementation. They include documentation, technical support, terms of use, and various operational aspects. Bringing transparency to the process through frequent discussions between architects, product owners, and engineering teams was crucial. Getting feedback from technical writers helped to achieve vocabulary consistency across APIs. For sure, all of the teams were aligned on what the success is: to build APIs that developers will love and want to use.

The road ahead

We delivered modern RESTful APIs that cover a subset of our overall marketplace capabilities and follow industry standards, well-established patterns, and best practices. Still, they are powered by a model that is flexible and extensible enough to capture new opportunities that might come in the future. Our journey is not yet complete. We are engaging customers, listening to feedback, and encouraging adoption of the new APIs, all to bring our new, long-term public API program to reality. Our goal is a large and powerful ecosystem of developer applications that add value and benefits to our buyers and sellers. Finally, we want to continue transforming our business by exposing valuable eBay solutions and capabilities to empower developers.