The public preview of the Azure Container Instances service brings cloud-hosted containers much closer to the idea of containers generally.

“Containers offer a new form of agility,” pointed out Corey Sanders, Microsoft’s Head of Product for Azure Compute. “You can ship apps significantly faster, you can update them faster, you can make changes and work together across a broad team in a much simpler way, giving apps and services much more agility. They also offer a huge amount of portability; you can build an app and deploy across multiple cloud and multiple app platforms. And because of their quick start time and very small size they offer really easy and rapid scale; you can scale your app nearly instantaneously to meet the demands of your customers or end users.”

If you’re experimenting with containers or working on a prototype, the need to first set up the VM that will host the container (and to remember to patch and delete the container later) is a speed bump for getting started. With ACI, you type in a command and get a running instance of the container image you reference — with a public IP if you want one — in less than five seconds. (If you don’t already have a resource group on Azure you have to create that, which adds another few seconds at the command line the first time.)

What you get back from the command line interface (CLI) is a JSON blob with the stats and status of your container — including the IP address. You can get more details with the logs command and remove the container with delete. There’s a Node.js Hello World application on GitHub and Docket Hub to experiment with.

Using the Azure CLI to spin up containers with ACI isn’t restricted to the Azure Cloud Shell in the portal; you can use it anywhere the CLI is available, which means on your PC or Mac or even your iPhone. The New Stack saw a demo of starting containers with the build of the Azure CLI that’s in the Azure Cloud Shell and Microsoft told us that “the new ACI container commands will be available in the downloadable Azure CLI within a few days.”

You can also deploy from a public Docker repository, from a private registry or from the Azure Container Registry. ACI isn’t limited to individual containers; you can deploy multiple containers as a container group that shares a host machine, local network, storage, and life cycle using a template. You can directly mount Azure file shares to persist and retrieve state with container instances and Microsoft plans to integrate with Azure’s virtual networks, load balancers, and other networking infrastructure.

Built-In Management

Azure manages the underlying infrastructure for container instances, like finding an appropriate host machine to run the container on, but for more complex container infrastructure you’ll still want to use an orchestrator to handle things like scaling and coordinated upgrades, Sanders said. “The way to think about Azure Container Instances is that they’re very much focused on simple container management, and enabling fast and easy deployment,” but they can also be building blocks for other container tools.

The open source ACI Kubernetes connector that Microsoft is working on, using container groups to create Kubernetes pods, is a sample to show how a whole container orchestrators can integrate with ACI using its REST API (which will be published in Swagger). If you want to use the Microsoft Draft Kubernetes app builder tool with ACI, the ACI Kubernetes connector enables that. The connector also lets you have traditional VM-based containers for predictable workloads and on-demand container instances for bursting and scaling, in the same Kubernetes cluster.

“You can use VMs for packing your workloads very tightly or you can use Azure Container Instances to deploy containers in a much more agile way, to be able to spin them up and deploy them down in seconds and to burst very, very quickly,” Sanders said, predicting that most organizations will use a combination of the two.

The connector is “still experimental and not ready for production use” but will allow you to manage container instances on Azure from your own Kubernetes environment (although for many workloads you’ll want to have containers on the same network). The connector can be used in any Kubernetes environment. The connector simply acts as a proxy to ACI, using an Azure service principal to create the necessary Azure resources.

The pricing for instances is also unusual; it combines per second and resource pricing. The default for an instance is a single CPU and 1.5GB of memory, but you can choose up to four cores and allocate between 1GB and 3.5GB to each core. Create requests cost $0.0025 and memory and CPU cost $0.0000125 per second (per GB or core). “This allows customers and developers to make sure that the platform is perfectly fitting their workload, not paying for second than more required or a GB of memory more than is necessary,” Microsoft asserts.

An increasing number of Azure services support containers (credit Microsoft)

Running a container non-stop would cost about $50 a month, although that’s not what the service is designed for; think devtest, rendering, scripts, experimentation and scaling instead. Although you can have a public IP address for your container instance, having a public endpoint optional; “You can also use this to run any sort of batch job workload,” Sanders said.

Azure container architect and Kubernetes co-founder Brendan Burns talks about “herdless cattle” uses, where you don’t need hyperscale but you don’t want to use your own infrastructure either. Running a script that executes quickly could mean a significant saving with per-second billing, and it may be a better match for some workloads than serverless functions, where you pay per request.

Because containers instances are a first-class object in Azure, not buried inside a VM, you can track usage of individual containers with billing tags if you want, and you can use Role-Based Access Control to determine who can connect to the instance.

ACI benefits from the Windows Server underpinnings of Azure, and in particular Windows Server 2016.

“Deployment is securely isolated between customers using technology from Hyper-V isolation,” said Sanders. “With Azure Container Instances, your application is as isolated in a container as it would be in a VM.”

First Class Treatment

Making containers first-class objects on Azure is a sign of how committed Microsoft is to containers, and how useful it expect them to be for customers. That means ACI isn’t just a “quick and dirty” way of spinning up containers. The flexibility and economics of Azure Container Instances make them a significant building block.

“We do expect some Azure services to build on top of Azure Container Instances and enable solutions using it and we are working with several of those right now,” Sanders told us. He didn’t name the services, but Container Instances would seem to be ideal for Azure Container Services, Azure App Service, Service Fabric and Azure Batch.

Feature image by Ashley Knedler via Unsplash.

The post Azure Container Instances Promises Cheaper, More Agile Container Tools appeared first on The New Stack.

With SQL Server 2017, Microsoft doesn’t just want to bring its heavyweight RDBMS to Linux; it wants to make it more accessible to developers so it can take advantage part of the trend to DevOps and continuous integration through containers.

That’s something SQL Server has been excluded from in the past; not just because it was running on Windows Server but because the way it’s been deployed and managed is so different from the other components developers work with, SQL program manager Tony Petrossian told the New Stack.

“People have been doing CI/CD and DevOps for building database applications for some time but SQL Server itself has never participated in that ecosystem as a component of the application because SQL Server was always a server managed somewhere else. As a developer, what you get is a connection string to a database somewhere in the IT world; the database binary itself was always disassociated from the app development process.”

Moving SQL Server into containers doesn’t just mean a much faster deployment; it means you can deploy it as part of development rather than operations — and test your database code like any other code. You can even put a sanitized subset of your data in the container for devtest and have the pipeline automatically test code with the full dataset in a secure environment.

“What we really wanted to do was allow the developer to pull the actual binary of SQL Server as a component of their application in their CI/CD or whatever other development processes they use, no differently from any other library they might have as part of their application,” Petrossian explained. “So the application and the database are tested together at the same time through all the app changes the developer makes, and all of the automation of the testing includes the database itself.”

That stops developers who want to use features in new releases of SQL Server which aren’t gated by operations and IT processes designed for the slower release cadences of enterprise software. “Ultimately we’re hoping this will simplify upgrades because if the developer decides they want a newer version of SQL Server because it has a particular feature then they can just pull it in and change their app to use the feature. Then all their automation and testing kick in and they validate the newer version of the database along with their app, as opposed to developers writing their new code and asking IT to go upgrade some server some place — which might take some time or not happen at all.”

“That distance is always very cumbersome and being able to do it all in one place will simplify the process for everyone,” he suggested. Even the IT teams who’ve helped create this divide aren’t happy with the outcome because developers tend to just work around them.

“What customers are telling us is ‘If I can’t upgrade and I can’t have the developers do what they need to do, they just pick up some other product off the internet and throw it in and pretend it’s just a library in their app and now we have to support this other thing’. They wanted us to make it as simple to throw SQL Server in there as it is to pick up any open source library that a developer may choose to include in their app.”

Containerizing SQL Server also improves the collaboration experience for developers who work on different platforms. “Because it’s running in a Docker container, you could run on it any versions of Linux, any distro,” he pointed out. “If I’m working on Ubuntu and my app eventually runs in Red Hat, or I pass my app on to my peers to add something to it and they’re on a different version of Ubuntu, it doesn’t matter because it’s all self-contained. All the dependencies are baked in, including the version of SQL Server. Having the exact environment in your development environment matching your staging path and production is no longer that critical.”

And when development is finished, deployment isn’t a different process. “You just publish the production version and these production versions of the containers can deploy into production.”

Developers on the SQL Server team who build tools are finding this improves their own productivity and changes their workflow in ways that will sound familiar to anyone used to DevOps. “Instead of pointing other developers to stuff they have to pick up and install, they say ‘go get this container that has the particular build of SQL Server and the version of the tool you need to get your stuff done’,” explained Petrossian.

“Pretty much nobody installs anything anymore; they tend to keep their machines clean and pure. It’s a lot easier to spin up a container, do your work and chuck it away than to pollute your machine installing stuff on it that you might not ever be able to delete. It’s another simplification and — now SQL Server is part of that.”

In particular, he noted, it gives developers more choice over their development environment.

“On the SQL team, we have a mix of people who prefer to run on Macs — people who work on cross-platform tools tend to run on Macs — and some who’ve been on Windows for a long time. This containerized environment means you can have SQL Server running either way and it’s just easy to pass things around now. Even in Windows, you can run Linux images in containers, so it’s becoming easy to maintain a unified code base with the same version of everything running on a different platform. We’ve been very careful about full compatibility across Windows and Linux, so customers can move a database between Windows and Linux whenever they want. You basically copy the files, mount them and SQL Server comes up. It’s not a migration or a port or anything; it’s like moving a database from machine A to machine B and they just have a different OS. And with all the Linux tooling in Windows now, it’s really easy to copy stuff around.”

If all of this sounds just like the usual DevOps advantage, it is, says Petrossian. “This is very much In line with what other people are doing in building applications, but it happens, in this case, to include a database.”

Feature image from Microsoft.

The post How Docker Containers Helps SQL Server 2017 Shift Deployment to the Developer appeared first on The New Stack.

The Windows Subsystem for Linux — the way that Windows 10 is able to run command line Linux binaries directly, without a virtual machine — is now out of beta. The company is also introducing WSL into the Windows Server environment as well.

“We’re adding WSL to Windows Server in response to the many requests we’ve received since launching WSL on Windows client,” Microsoft’s Rich Turner told the New Stack. “Many DevOps engineers — systems engineers, administrators, and developers — want and need to deploy, manage, monitor, and maintain apps and systems that incorporate tools that run best, or only, on GNU/Linux platforms. They want to run local services to host, for example, databases and websites for local development and debugging. Being able to run one’s preferred Linux distros and tools atop WSL on Windows Server will enable dev-ops engineers to get more done on Windows Server.”

Turner describes “a very lightweight Linux runtime environment for local interactive user scenarios.” As part of the Fall Creators Update (1709) of Windows 10, WSL will be fully supported, so you can file issues through your standard Microsoft support channels (and use the Windows 10 Feedback Hub as well as the WSL issues GitHub repo).

The process for installing a Linux distro changes as well; distros for WSL now come from the Windows Store. Ubuntu, openSUSE Leap 42 and SUSE Linux Enterprise Server 12 are already available and Fedora is coming soon. Moving the distros to the Store simplifies installation and speeds up downloads (by using the Store’s content distribution network). It also means that you can run different distros simultaneously on the same machine, without reinstalling or rebooting.

Use the Windows wslconfig command to manage which distro you’re working with. The /l flag lists your installed distributions; use /setdefault <DistributionName> to choose the default distro that starts when you type bash at the command prompt. You can also type ubuntu or fedora to launch the distro you want (or use their tiles on the Start menu) — and you can use those commands in scripts to run tests across multiple distros automatically.

If you already have Ubuntu installed with WSL it won’t stop working, but you will still want to switch to the Store distributed version because distro updates will now come through the Store and at some point, the directly installed version will be deprecated. To do that, back up any files and config information you want to keep (for example make a WSL-backup folder in /mnt/c/tmp/WSL-backup/ or in your user directory (which is /mnt/c/Users/<Username>/Desktop/); if you’re moving a lot of files, tar them first. Uninstall your existing Ubuntu using the lxrun /uninstall command from the Windows command prompt or PowerShell, then install Ubuntu from the store and copy or untar your files into the new Ubuntu instance.

“If you’re moving settings between instances of the same distro family, the move should be relatively simple,” Turner notes, but he suggests using extreme caution if you’re using a different distro because the configuration systems and file locations aren’t always the same; it’s probably best to just create your settings from scratch rather than trying to save time by copying them to multiple distros.

WSL is also coming to Windows Server, as part of the Windows Server Insider program. That doesn’t change the reason WSL is in Windows; it’s still for developers using command line tools and integration with toolchains, not for running production workloads or server applications like Redis (even though they might work). And it’s not intended to replace PowerShell, which is still the way to manage server workloads and Windows Server itself.

That said, Turner also cautioned that “WSL is only for user-initiated interactive tasks, not for production service workloads. WSL was specifically built for users to run tools, locally build, deploy, and debug software, modify configuration, monitor workloads, and so on, but should not be used to host production daemon/jobs, or server workloads like Apache, Nginx, MySQL, PostgreSQL or MongoDB.”

There are performance as well as support reasons not to try using WSL for server workloads. A recent set of independent benchmarks [www.phoronix.com/scan.php?page=article&item=windows10-wsl-scaling&num=1] suggests that while general performance isn’t far off running Linux directly on the same hardware, I/O performance continues to lag (it’s an area where Microsoft is still promising improvements).

Setting up WSL on Windows Server

To start using WSL on Windows Server, make sure you’re using a Windows Insider preview build and run this PowerShell command (as administrator) from PowerShell:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

The same WSL distros you can use on Windows 10 will run on Windows Server and they still come from the Windows Store on Windows Server, but you have to manually download them from the Store CDN, with the Invoke-WebRequest cmdlet. Ubuntu is at https://aka.ms/wsl-ubuntu-1604, OpenSUSE is at https://aka.ms/wsl-opensuse-42, SLES is at https://aka.ms/wsl-sles-12 and Fedora is coming soon. To download Ubuntu, the PowerShell command is:

Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1604 -OutFile ~/Ubuntu.zip -UseBasicParsing

Extract the distro to a target directory on your system drive. For example:

 Expand-Archive ~/Ubuntu.zip ~/Ubuntu

Now run the installer, which is named for the distro (ubunte.exe, fedora.exe and so on), create your UNIX user and set the username and password in the same way you do for WSL on Windows 10.

Hyper-V Linux containers give you the isolation of Hyper-V with the flexibility of Docker containers and you can manage them with Linux tools now that WSL is coming to Windows Server (credit Microsoft).

If you’ve been using WSL on Windows 10 and you want to save time by copying your settings across to Windows Server, you can do that the same way as on Windows 10 — but again, stick to copying your config from the same distro to avoid problems.

Mixing Containers

WSL will be Windows Server particularly interesting if you’re running Linux containers on Windows Server using Hyper-V Containers. Think of them as midway between a container and a VM; they run in a lightweight VM with minimal services, so they have their own OS and kernel, but you don’t have to manage the VM — you just manage the Docker container.

As Microsoft’s Taylor Brown explained to us, “WSL for Windows Server is a great way to use your existing deployment scripts and tools with the container technology we’ve brought to Server. If you want to run Linux containers, here’s a way to run your Bash scripts to build and launch and debug and deploy and do Linux native things using Linux containers on Windows Server..”

That’s in addition to the Windows format Docker containers in Windows Server 2016 and Windows 10 because Microsoft customers use both. Brown encourages developers to experiment with the cross-platform. “We’ve been brainstorming what you could do with. Now that I can run Docker client in WSL if I want to and all the tools around that just come across, and now that I’ve got multiple stage builds, could I actually use a Linux container to build my Windows image or a Windows container to build my Linux image? The answer is probably yes.”

The post Microsoft’s Lightweight Linux Runtime Environment Comes to Windows Server appeared first on The New Stack.

Microsoft is turning just about everything in Azure into events you can connect together with serverless code.

Serverless computing lets you focus on the app you’re coding rather than the infrastructure that it will run on; you write your app as functions triggered by events, so that you only pay to run your code when there are events for it to process, and the service takes care of scaling up to handle the events that trigger your code.

Handling those events can still mean caring about some infrastructure, like the message queues that you have to interact with. That’s where Microsoft’s new Azure Event Grid service comes in: it’s a fully managed event routing service that lets you subscribe — and react — to any event happening across other Azure services, as well as events from other services that you monitor via webhooks and REST APIs.

“We’re seeing many more modern apps being built around events, whether it’s reacting to changes coming from IoT devices whether it’s responding to user clicks on mobile applications or initiating business processes from customer requests,” Corey Sanders, Microsoft’s head of product for Azure Compute told the New Stack.

Events are becoming the same first-class, native object across the full range of Microsoft services.

“Serverless platforms, like Azure Functions and Azure Logic Apps, allow you to use on-demand event-based application models without any of the weight or baggage of the infrastructure. But as we see more customers deploy apps this way, we’re seeing increased challenges in other aspects of the application model — primarily in managing and responding to those events that are initiated across many different platforms,” Sanders said. “Events come from a plethora of sources and, until today, they’ve had to be managed with lots of additional code. Event Grid completes the story; it’s a single managed service for managing all events from any source, to any destination, that’s built to be highly available and handle massive scale dynamically. It’s the second half of what serverless applications really need — a reliable global eventing solution.”

Azure already has various ways to use events: in Functions and Logic Apps, in Power Apps for building custom business apps, in Azure Event Hubs for logging telemetry from apps and sites, in Azure IoT Hub for events from IoT assets and even in Cosmos DB with the new Change Feed processor. Event Grid gives you one central place to manage the routing of those events, using filters to route specific events to different endpoints (or even multicast to multiple endpoints).

It’s built on Azure Service Fabric, which powers key Microsoft services from SQL DB to Power BI, which means that events are becoming the same first-class, native object across the full range of Microsoft services, starting with Azure (and that the service can cope with events at very large scale). That makes it very different from other services that handle events but only do it through REST APIs and webhooks.

“These events are completely decoupled from the app platform,” explained Sanders. “Event Grid is more than just a standard pub-sub system because of the breadth of services covered by the platform, but also because events are first class which means you have direct control over them, including filtering around prefix or event type. you can control and configure a set of filters so that you can easily decide which events to listen and react to in your code. That filtering is done on the server side and that means the complexity of code you need to decide which events to react and respond to is dramatically reduced.”

Filtering by event type and prefix let you precisely isolate the events to respond to, even from a very large stream of events.

Events Go across Azure

The obvious use for Event Grid is to route between different Azure services; uploading an image file to Azure Storage could trigger an Azure function that resizes the image or runs image recognition on it. “Imagine a custom event being fired from an IoT-based solution and launching a function or a third-party eventing service,” Sanders suggested.

Event Grid is a perfect match for Azure Functions, but it doesn’t only work with serverless services. “The goal is that it becomes a central point for developers and system administrators to use for managing all their events, and from every service that’s running in Azure so customers don’t have to manage multiple services multiple sources, multiple different types of platforms and can enable all this in a single, simple way,” he explained.

“We have plans for the broad set of services to be integrated, including Cosmos DB, Azure AD, SQL DB, and Postgres and MySQL. We’ll offer direct IoT Hub and Event Hub integration with Event Grid to be able to take those messages, the storage and the managing of those messages and be able to convert them into these first-class eventing objects that then are routed properly to the service or scenario you want to execute.”

In the preview, you can collect events from Azure Resource Groups and Azure Subscriptions (for management operations), Event Hubs and your own custom topics that you create. Blob Storage and Azure Automation support is coming soon, and Azure Active Directory, API Management, Logic Apps, IoT Hub, Service Bus, Azure Data Lake Store and Cosmos DB will join the list “later this year.”

Initially, you can send the events to Azure Functions, Logic Apps (which covers Power Apps and Microsoft Dynamics, as well as third-party services like Salesforce), Azure Automation and webhooks to external services. Later on, you’ll be able to route events to Service Bus, Event Hubs, Azure Data Factory and Storage Queues.

You can also use Event Grid for operations and automation; when it launches the service will support Azure Automation from Azure Operations Management Suite. “You’ll be able to be able to take an event from any source and be able to run a script or response as part of that, and the alerting and monitoring systems will be integrated over time,” Sanders told us.

That means you can track how you’re using Azure services like VMs and SQL DB, to enforce policies or automate administration by putting metadata into operations tools or creating work items. “With a few lines of code, a virtual machine being created could result in a full app workflow that adds a billing tag and sends a notification that the VM has been created. That end-to-end capability is very powerful for managing control and access to the environment and in future, perhaps including Azure Active Directory changes in addition to the API changes.”

Event Grid in Action

Event Grid is designed for large scale. “We’re thinking in millions when comes to event-based management and communications,” he said. Microsoft has already tested millions of events per second in a single region, sustained for several days and that’s how the service is priced – at $0.60 per million operations ($0.30 during preview) with the first 100,000 operations per month free during the preview. Operations are defined as event ingress, advanced match, delivery attempt, and management calls.

There are other limits during preview: manage operations are throttled to 10 per second and you can have 1,000 event subscriptions per account. When the service launches, expect to see 10 million events per region and 100 million event listeners supported. “The goal is to really offer that fast transmission of events through the system,” Sanders explained.

If you’ve used Azure Logic Apps, the visual programming interface for defining events with triggers and actions will look familiar (although with the option to flip into code view to extracting details from your Azure subscription).

Event Grid has predefined common triggers or you can define your own.

Events have common information like the event source, the time it happened and a unique identifier, as well as specific information for the event. The metadata for a file saved into Azure Storage, or if the event is a VM reboot it will include the VM name and reason for rebooting.

The schema of what information is in each event comes from the event topic, and creating custom topics is how you define the endpoints that events are posted to. Event Grid topics are Azure resources and you create them in Azure resource groups. Subscribing to a topic is how you tell Event Grid which events you want to track; subscriptions are how Event Grid routes and filters events.

The event handler (which Event Grid calls a subscriber) is the final destination of the event and the actions to be taken to process it — which includes retrying for up to 24 hours, with an exponential backoff time, until the destination service confirms the event has been accepted.

Subscriptions are how Event Grid filters and routes events

The long list of actions for events builds on the many connectors Azure already has for serverless computing.

Event Grid might look like an extension of existing Azure event handling, and having a single, central point for handling all your events in Azure is certainly convenient; but the fact that it’s built on Service Fabric and thus plumbed deeply into the way Azure services are built makes it more significant than that. Microsoft is taking some of the core principles of event-driven, serverless computing and using them to interconnect disparate Azure services — and anything else you want to receive events from and send events to, giving you a simple but very scalable way to create serverless apps that span services and platforms.

Feature image via Pixabay.

The post With Event Grid, Microsoft Provides the Missing Half of Azure Serverless Computing appeared first on The New Stack.

Automated testing is increasingly important in development, especially for finding security issues, but fuzz testing requires a high level of expertise — and the sheer volume of code developers are working with, from third-party components to open source frameworks and projects, makes it hard to test every line of code. Now, a set of artificial intelligence-powered options like Microsoft’s Security Risk Detection service and Diffblue’s security scanner and test generation tools aim to make these techniques easier, faster and accessible to more developers.

“If you ask developers what the most hated aspect of their job is, it’s testing and debugging,” Diffblue CEO and University of Oxford Professor of Computer Science Daniel Kroening told the New Stack.

The Diffblue tools use generic algorithms to generate possible tests, and reinforcement learning combined with a solver search to make sure that the code it’s giving you is the shortest possible program, which forces the machine learning system to generalize rather than stick to just the examples in its training set.

“What we have built is something that does that for you. If you give us a Java program, which could be class files or compiled Java bytecode, we give you back a bunch of Java code in the form of tests for your favorite testing framework,” Kroening said.

The results can also be used as JUnit tests with some suitable mocking framework.

Diffblue started with Java because test automation is so common in the JUnit community, but it is adding support for Python, JavaScript (starting with Node.js), C# and eventually PHP. For all the languages, the tools can work with both source code and binaries. You can keep the generated tests so you can keep using them through your development cycle. The security scanner also provides advice. “If it finds a SQL injection flaw, it will give you the input strings that trigger the vulnerability,” Kroening explained.

Today the product is a web service running inside a Docker container, and you give the service the URL of your code repo. That makes it easy to integrate with an existing DevOps process. “If you have something like Jenkins set up it fits in there extremely well,” he noted. “In the longer term, I envision a lightweight environment that runs on a laptop, inside an IDE; think of it being integrated into Eclipse or Visual Studio. It’s extremely valuable to provide feedback immediately after you’re written a piece of code, not the next day after the Jenkins build has failed.”

Fuzzing It for You

Microsoft Security Risk Detection (previously known as Project Springfield) takes a slightly different approach. It’s a suite of fuzzers that find security bugs in Windows and (currently in private preview) Linux applications, and an automation platform that tests your binary with the many inputs these fuzzers create, in an Azure virtual machine.

“We’re packaging up some AI that we’ve been using inside Microsoft for over a decade into a cloud service that customers can use to spin up testing labs on demand and use them on the binaries they are about to ship,” Microsoft security researcher David Molnar explained to us. The idea: “How we can help people use a service on their binaries, on code they’re shipping and code they’re buying, on code that runs their business, that really helps them find bugs that they couldn’t find before?”

The AI in Springfield combines two techniques; time travel debugging and constraint solving. Molnar compared time travel debugging to working out how a car crash happened. “If you see a car crash, you can try to figure out what happened by looking at how the fender is bent and the way the wheel is turned or the state of the windshield, but you can only learn so much that way. Now imagine you had a video of the two minutes before the car crashed and you could step through it frame by frame. We can do that with your program as it’s executing. We combine that with a form of AI called Constraint Solving which allows us to ask ‘what if’ questions. I want to know, what do I need to change about my input to increase my coverage and make my if statement true?”

The ideas behind solvers go back to the 1970s but originally they only worked on small examples, he explained. “Now we can solve questions quickly and on a scale that would have seemed unimaginable before; we’re able to handle dynamic instruction traces of over a billion instructions.” The portfolio of fuzzing techniques in the service has also been developed over many years (the name comes from the dial-up line noise that inspired the first fuzzing techniques of crashing software by injecting noisy inputs).

“We’re adding more fuzzers on the back end and exploring how to increase the efficacy of the fuzzers,” Molnar said; there are new techniques both from Microsoft Research and the open source world that may make their way into the service in time.

Molnar is the researcher running the team behind Springfield; previously he helped apply the same techniques to products like Windows and Microsoft Office, finding a third of the security bugs discovered by fuzzing in the Windows 7 client. That required a lot of trial and error and hands on work, as well as expertise in fuzzing.

“Now we’re about to able to hide most of that behind a GUI; you don’t have to turn any knobs but under the hood,” Molnar said.

You do need to have some initial test cases for the service to start with, he explained. “Those can be the same functional or regression tests you might have lying around, or you can search the web to find some initial test cases.”

The information produced by the service is designed to be useful for testers as well as developers, so it does more than just find bugs; it also helps you triage them. “We make sure the bug is reproducible; we repro it five times. We have a part of the system that looks at every crash and tells you what kind of issue it is and if we think it’s exploitable or not.”

He called the technique for deciding that “a best effort” but suggested that the service gives even security experts who are familiar with fuzzers a level of deduplication and triage that simplifies their task.

Running the service over your code means uploading the binary to a virtual machine that you can install any necessary pre-requisites on. “What we like about giving customer a full VM is that we already see customers who have dependencies. If we told you to just upload a zip or give us access to your source repo there are customers who wouldn’t be able to use us, who can use us because it’s a VM. If you need to install SQL or a kernel driver or make crazy registry key modifications [for your software], you can,” he said.

Because the service works on binaries, you can use it on third party code where you don’t have the source. “You might have a DLL that your company has built over many years or you might want to test new software that you’re thinking of using. Some customers say ‘I have server-side code powering my business and forget about security, it just can’t go down, I don’t want anything that can cause it to fall over.’ You don’t need to know the details of the library or the source code or how to call it. When we used this service on Windows, the Windows team wrote a wrapper around the libraries that make up parts of Windows but they didn’t have to write annotations or explain the source code to us.”

The disadvantage of using a VM is that it’s harder to build into existing DevOps and testing workflows. It would be an ideal companion to entity repositories like Sonatype and WhiteSource that help you track what third party code you’re using as part of your development. That’s something Molnar hopes the service will be able to do in the future.

When you find bugs in third party code that you can’t change, he suggested a range of mitigations, like sandboxes, virtualization, EMET and Exploit Guard in Windows 10. “That’s why we call the service risk detection — because we help you figure out what the risk is, then offer a whole range of options to contain it.”

Beyond Bugs

Some researchers are trying to go further than just finding bugs to fixing them. The new Accurate Condition System from Microsoft Research, Peking University and the University of Electronic Science and Technology of China uses a combination of natural language analysis and statistical analysis of open source code available online to generate patches that aim to fix the underlying bug, not just pass a test suite.

Kroening also hopes Diffblue will eventually create code too. “The ultimate goal is to enable programming to be something that can be done by non-experts. I want to enable you to produce code from scratch, not debug existing code, by means of a learning engine.”

Feature image by Jairo Alzate on Unsplash.

The post Machine Learning Lends a Hand for Automated Software Testing appeared first on The New Stack.

If you’ve come across the Okta set of cloud identity services, you’ve probably thought about it as an IT tool. Administrators can use it to manage logins to multiple cloud services like Office 365 and Google Cloud and Salesforce through a single interface, complete with policy on password strength and who can get access.

But Okta is also making more of its authentication and user management available through APIs so developers too can build these services into their apps, much like they would with API-driven identity services like Twilio’s Authy, Auth0, and Microsoft’s Azure Active Directory B2C.

With data security so important (and data breaches caused by poorly managed credentials so ubiquitous), it’s time to stop building poorly encrypted user and password systems (or copy and pasting code from Stack Overflow that does nothing more than base64-encode the password). Get a password reset system wrong and millions of account details can be lost. Just as Twilio APIs for messaging and communications and Stripe or Braintree APIs for payment, identity management is becoming a commodity it’s better not to write yourself.

“The urgency of everyone having to build customer web applications and take their products and services online mean there’s a big push to deliver and there just aren’t enough developers to do it,” Okta’s vice president for the developer platform Alex Salazar told the New Stack. They go for the path of least resistance and that is not always the best, most secure, most maintainable option; we’re trying to change that by having a good developer experience.”

That’s not just about the technology in Okta’s service, he maintained, but also the APIs and documentation for them. “We see documentation as product; we can’t announce [something new] until there’s documentation. It’s the same with SDKs; if something isn’t in the SDK and framework, it doesn’t exist,” Salazar said.

More Factors, More Complications

Passwords and usernames aren’t enough for security and user management for your apps these days. Increasingly, users require multi-factor authentication, which, for an increasing range of devices can be quite complicated to deploy. Even developers who turn to open source software such as the OAuth specification and the OpenID Connect protocol to simplify secure connections face an uphill task, Salazar said.

“If you know anything about identity, you probably know about OAuth. The problem is that implementing standard OAuth is actually not the best path. Even if you’re using fully compliant libraries that are maintained by a major vendor, you’re still missing out on security best practices most developers don’t know about,” Salazar said.

OAuth uses tokens to make it harder to intercept and replay credentials, and those must be used correctly; like not using an identity token as a session token because it might contain personal information, or not using an embedded web view in a native app to authenticate users because then they have to re-authenticate every time they use the app, because that embedded flow runs in a sandbox that doesn’t know they’re already logged in.

Developers also need to know how to handle flows like an assurance about how users have authenticated, and token revocation, he pointed out. “If your app is using tokens and you’ve got a delinquent or malicious user, you want to turn them off. The OAuth spec doesn’t talk about that so now you have to build out your own custom scheme of token revocation,” Salazar suggested.

Often, developers don’t even know this is a problem until it rears its head in production and then many don’t really have standard patterns on how to remediate the issue.

To help with that, when a developer signs into Okta they can pick what kind of app they’re developing and get templates and quickstarts that show what steps are involved for secure registration, email verification and password reset (and there, workflows are being added).

“We try to put you on the right path to make the best choices without you having to do too much extra work,” Salazar told us. “We’re targeting the most popular backend and front-end languages.” Libraries for Node/Express, Java/Spring, and ASP.NET to support developers building applications that run on the server, as well as for Angular and React on the front end side, and for iOS and Android as well. For other technologies, Okta offers documentation on using the APIs.

Okta has also created an embeddable, customizable JavaScript sign-in widget using its APIs that you can use to get an end-to-end sign in code flow, including multifactor authentication, password expiry and self-service account recovery that supports social authentication providers.

Another big move is that Okta is no longer just for user management in first-party apps; it can be used for customers as well as internal users, and it will soon support developers writing third-party apps. So if, say, a solar roof installer that uses Okta for customer identity has so many users that other developers want to write an app for them to use, they’ll be able to use those Okta identities in their app. Okta is also integrating with a wide range of cloud email services and OpenID Connect providers.

Multifactor authentication is now included in all Okta subscriptions, with one-time passwords delivered by email or along with detecting common passwords and blacklisting of IP addresses known to be malicious. A service like Okta lets you offer multifactor authentication without having to have your own infrastructure, and the quick starts are designed to make it easy for developers to implement as well.

“The problem with multifactor authentication is that it’s complicated,” Salazar said. “It needs to be more convenient; if you don’t make it easy for people to implement, it’s not going to get used. It also needs to be a good experience. If it’s not a good experience no one turns it on, and you need users to adopt multifactor authentication. Too many developers grab something insecure and don’t make it easy for end users.”

The next step is “passwordless” authentication, which services like Yahoo and Google already offer. One Australian insurance company is already using Okta to do that; because customers don’t log in enough to remember a password, they just use their email address to login — triggering a test message with a one-time token. Okta is working on a workflow for developers to deliver that. “Over the next 13 months, we will deliver a solid passwordless experience to customers, and also make sure it’s secure,” Salazar told us.

The post Okta Wants to Be an Identity Service for Developers appeared first on The New Stack.

A growing number of machine learning toolkits, especially for deep learning, have become available recently for developers; Google, Microsoft, Amazon, Facebook, Samsung, Intel and Baidu all have released their own, plus there are key frameworks like Torch, Caffe2 and Theano, and newer frameworks like Marvin that have come from academic groups.

Different machine learning teams use different frameworks, especially as they all have different advantages and disadvantages, but until now, there haven’t been many options for moving from one framework to another without completely recreating your data model.

Reproducibility is important for researchers who want to compare and replicate results published by other teams who use different tools or programming languages, as well as for developers who want to prototype a data model in one framework and then use a different toolkit in their production system. That usually means throwing away the work that’s done and recreating it in the new framework, which slows researchers down just at the point that they’re ready to roll out your machine learning system.

One way around this is to use Keras, a Python library that offers high-level building blocks for developing deep learning models using TensorFlow, Theano, Deeplearning4j and the Microsoft Cognitive Toolkit (CNTK); The same model can be used with different frameworks just by setting flags, without changing any code. Testing with an early version of the CNTK integration shows that the different frameworks give better results in different areas. Keras is designed to be more user-friendly than the frameworks themselves and it’s excellent for prototyping and experimentation.

It isn’t as good at scaling past a single-node machine though, which is increasingly the direction that deep learning is taking (to scale out larger, deeper networks); you have to add in other tools like Elephas to do that effectively. Having the Keras API acting as an abstraction doesn’t always provide access to the more powerful options in the underlying toolkits, without tweaking the Keras framework itself.

The Open Neural Network Exchange (ONNX) format that Facebook and Microsoft have collaborated on takes a different approach. It’s an open-source standard for representing deep learning models that will let you transfer them between CNTK, Caffe2 and PyTorch.

Unlike Keras, ONNX is just standardizing the way the data model is represented. Each framework can be used directly, but the model can be reused across different frameworks.

It’s common for machine learning projects to use multiple frameworks at different stages of development; one framework may be optimized for fast training — great for prototyping ideas — while another works with different network architectures so it is more suited for building a final product. Now that smartphones often have an “AI core” in their systems-on-a-chip, more learning can be done on devices, so a framework like Caffe2 would be a choice, given that it is optimized for inferencing on the phone. ONNX makes it easier to move from one to another.

Disconnects

“The main intent of Microsoft and Facebook’s work on ONNX is to address and (hopefully) repair the disconnects that can occur when AI projects move from dev/test into production,” Pund-IT Principal Analyst Charles King told The New Stack. “Support for data/model interchangeability is critical for developers hoping to create applications for multiple platforms since major vendors are also moving forward with their own proprietary AI frameworks (Apple’s Core ML is a good example),” King noted.

“It’s a nice way to bring different framework together so we can exchange models, because every framework has its own strengths and weaknesses and this will free people from being tied to one,” Microsoft Technical Fellow Xuedong Huang agreed.

“CNTK is fantastic for massive scale deep learning; if you have a massive amount of data, it’s the fastest. Caffe2 is very good for creating small models for computer vision workloads. PyTorch is very good at giving you the flexibility you need in research; it’s slow but it has the flexibility. When you move from early-stage research to massive industrial research, you might need to switch from PyTorch to CNTK, for example, and with ONNX, now you can easily do that.”

ONNX is also convenient for silicon vendors who are increasingly adding optimizations to speed up neural networks; Nvidia’s Deep Learning SDK covers a wide range of frameworks, but the more frameworks share the same representation for data models, the less work it is to optimize for them.

Inside ONNX

The latest versions of Caffe2 and PyTorch already have ONNX support and the next release of CNTK will add it. Microsoft and Facebook are also working on reference implementations, tools and a “model zoo” of model configurations that you can use to get started on a machine learning project quickly.

Many deep learning toolkits use computation graphs to represent neural networks, but they’ve always had their own graph formats. ONNX has a definition of an extensible computation graph model that all three frameworks can use, as well as definitions for standard data types and built-in operators. The ONNX computation graph is a list of nodes with inputs and outputs. Each node calls an operator; the operators aren’t stored in the graph, but each framework that supports ONNX will implement the operators for the data types the standard supports.

The first version concentrates on the operators and types for inferencing, but recurrent neural networks are a high priority for the PyTorch team working on ONNX support.

ONNX isn’t the only attempt to simplify the task is using multiple frameworks, King points out. The Predictive Model Markup Language (PMML) supports model interoperability in traditional neural networks, like Backpropagation. A number of companies, including Amazon, IBM and Google support and utilize PMML. The group that created PMML is currently developing a successor technology, Portable Format for Analytics (PFA) that is more flexible (using JSON instead of XML) and also handling data preparation along with the models themselves.

“ONNX should benefit a range of AI and associated machine learning (ML and deep learning (DL) processes, especially if it grows beyond the initial support,” King said. “Whether Microsoft and Facebook’s ONNX eventually emerges as an accepted standard remains to be seen but the two companies are heading in the right direction.”

Microsoft is a sponsor of The New Stack.

Feature image via Pixabay.

The post Facebook, Microsoft Bring Interoperable Models to Machine Learning Toolkits appeared first on The New Stack.

Cloud is a good fit for Internet of Things (IoT) because it provides both scale and connectivity to handle lots of sensor readings from large numbers of devices in different locations. The cloud, through platform services also offers tools for processing the information.

But the cloud can also be pretty inconvenient for IoT when the devices being monitored and measured are in remote locations without much connectivity; predictive analytics in the cloud triggered by anomalous sensor readings isn’t much use if the latency is so high that whatever the system trying to predict is going to happen before the data and predictions can get sent back and forth.

Microsoft’s Azure IoT Edge service (currently in preview) uses Docker containers as a way of packaging up key parts of the Azure IoT Hub cloud service to run on anything from a local server cluster to a tiny IoT board like a Raspberry Pi used as IoT gateways. That’s going to work with Azure Functions, using serverless computing locally to trigger actions, as well as taking functionality from Azure Machine Learning and Stream Analytics to use machine learning locally to analyze a stream of sensor data and send alerts.

In the future, that will also include Cognitive Services, Microsoft’s pre-built and customizable deep learning APIs for tasks like vision, text and speech recognition, Azure IoT Director Sam George told The New Stack. “IoT on the edge is already being used right now but people are having to develop their own custom solutions. What you want to do is to take that intelligence that’s available in the cloud and be able to run it on an IoT device directly, You’re able to take cloud intelligence, so you don’t have to custom develop intelligence for the edge, you can simply move it up to the devices.”

Microsoft has been adding data model offload to its Azure Machine Learning service, including for iOS 11. With Azure IoT Edge, the Azure monitoring portal will have options for exporting Docker containers from Azure ML, Functions and Stream Analytics and connecting them up to create sophisticated systems.

Schneider Electric is using Azure IoT Edge with the SCADA systems it sells to monitor remote oil wellheads, which run unattended. “Many of them are in remote locations where they only have a satellite uplink; they use that to send SMS saying ‘the pump broke’ and someone drives out and fixes it,” George explained. With Azure IoT Edge, the company attaches an IoT device to that SCADA system that’s running local machine learning to monitor the telemetry coming off the SCADA system. When the data suggests the pump is going to break in the future and the maintenance team can get someone out there to fix it before it does.

The Schneider Electric system for artificial lift pumps uses Azure Machine Learning and Azure IoT edge to bring predictive analytics to the edge.

“Containers are a natural way to allow not only portability but also reusability so that you can assemble just the right application,” George explained. “Azure IoT Edge apps are assembled from one or more containers and the containers do inter-container communications using the local message broker in IoT Edge.”

While one container is running an adapter to talk to the SCADA system, another container cleansing the data and transforming it into something the machine learning model expects. The company doesn’t currently use Azure Service Fabric orchestration system, “but in future, we will make Service Fabric options available for multi-node” George told us.

Choose Where to Run

The compatibility of containers provides a lot of flexibility for deployment within the Windows/Azure environment. The same machine learning models that run on Azure and can be deployed to IoT Edge devices will also run on Azure Stack, George pointed out; and they will also work on the Machine Learning Server that’s built into SQL Server 2017 (this is also available separately as the latest version of what used to be called Microsoft R Server).

Using containers also makes IoT device updates more efficient, George explained. IoT Edge keeps a “device twin” for each device so that the latest information from a device is always available even if it’s currently offline (which developers can carry on coding and testing with devices that aren’t connected all the time). It also has a module twin that controls the configuration of each container deployed through the service, whether that’s a machine learning model or an Azure Function.

“When you take a machine learning model that has input variables you can adjust, typically in machine learning you’re adjusting those continually. With Azure IoT Edge, you can not only place that machine learning model on the edge but you can remotely control it with the module twin; if you want to change one of the input variables you can set that in the cloud on the module twin there and we’ll replicate that change down, the machine learning model in container will get notified and can adjust its input variable, and it can turn around back and write back to the module twin so we can see that container has acknowledged and it’s now running in that state,” George said.

“It’s not just about placing workloads; it’s about managing and monitoring them when they’re there. You don’t want to send another half gigabyte model down just to adjust an input parameter.”

Azure IoT Edge isn’t the only way the Azure IoT services use containers; the remote monitoring tools in Azure IoT Hub include some preconfigured solutions. Customers like how easy those are to get started with, George told us but found customizing them hard, so they’ve been rewritten as microservices, both for .NET and Java. “The microservices run in a Kubernetes cluster and you can simply deploy updates to that, so it helps with agility. And If you want to replace part of the solution you can just replace that microservice.”

Microsoft is a sponsor of The New Stack.

Images via Microsoft.

The post Azure IoT Edge, Machine Learning and Containers appeared first on The New Stack.

Serverless can be a confusing name for event-driven computing because it focuses so much on what you don’t have to care about rather than what you can do. The name does serve as a reminder that serverless platforms don’t help manage state; and while it’s good practice to externalize state so that it doesn’t interact with the business logic of an application, all but the simplest applications have state that needs managing.

The new Durable Functions feature in Microsoft’s own serverless service, Azure Functions, is one option for managing state, and Chris Anderson, senior program manager for Azure Functions, told the New Stack he hopes to see other frameworks and tools for helping with state so that can stay separate from the business logic. After all, being able to focus on that logic is the real advantage of serverless.

As well as taking away the burden of managing infrastructure, serverless also takes away existing tools for solving problems like state. “When we gave you just functions, we took away the lot of things you used to have like frameworks to manage state. This is like trying to create a web server without a framework like ASP.NET so you’re having to manage every part of the workflow of each request, only across many different services. Durable Functions is a first step towards doing more than just a point-to-point trigger.”

Durable Functions addresses the task of managing state for an application. “What customers had to do before was manage all that external state themselves, by messages along on a queue and having a record in a database somewhere with the details of the job, which can be large amounts of data. The queue itself can’t hold a whole lot of data; it can only hold a pointer to a place where this is more data.”

Orchestration Patterns

Durable Functions are intended to address a variety of patterns and scenarios that would quickly get complicated using triggers, timers and notifications, especially when orchestrating a range of activities with a complex set of tasks that need to happen each time a particular event occurs.

The simplest case is a “straight line” of events. “When I have one task, that causes another task to occur, that causes another task to occur, that causes another task to occur… with some if statements and other business logic to fork the workflow, but I’m trying to go from point a to point b,” Anderson explained.

Fan-out and fan-in is another common serverless pattern. Think of how MapReduce works. “Fanning out work to many parallel nodes works great in serverless because you’ve got lots of tiny machines; so if you’ve got a big work item, you want to split it up into as many small nodes as possible and fan out, and serverless will keep on fanning out as much in parallel as you need to and then scale back in. Whether you have one big job or lots of big jobs, it right sizes itself so you can always reach maximum values of parallelism,” Anderson said.

The part that’s always hard is the “fan in” step, Anderson said, because the workflow needs to know when all those messages have completed, so it can go run the next step of the process. Durable Functions allows the developer to express that with a simple a FOR loop and an AWAIT call to wait for all those things fanned out to complete.

“What used to be lots of complex pieces like polling and error cases and so on can all now be expressed in five lines of C#,” Anderson boasted. And don’t think of that just for data parallel operations; it’s also useful for automation, like provisioning multiple VMs in parallel, with multiple steps that have to complete, including having someone sign off on the process.

Durable Functions can keep track of state in looping processes, like a game show with multiple rounds. “I can manipulate it to do things that create what I call durable stateful singletons, which is orchestration that runs in a loop that keeps track of the current state of that object, and I can send it external messages to make it do other things.”

Another complex scenario Durable Functions can help with is waiting for some external interaction. “In the real world, we’re going to have other external systems that interact with our orchestrations; I can always try to wrap those things up in other functions but I don’t want to have a function that’s polling some external service to know whether it’s there or not,” Anderson said.

Durable Functions supports the idea of an external event, whether that’s another service or another function elsewhere, calling into the durable functions to pass it along with a piece of information and Durable Functions can wait on that event.

State Needn’t Be Expensive

Anderson suggested that would be a good way to handle user interactions with something driven by functions, like a two-factor authentication (2FA) service. “Humans don’t operate at the same speed as restful services! It’s going to take a few seconds to take out the phone to see what the 2FA code is.”

The Durable Functions orchestration can send the notification to the user that there’s a 2FA code to input, wait for the code to come back from the user, validate it or allow two or three more attempts before killing the login attempt, all without keeping the function running — and charging you for it.

“You don’t have long running functions waiting for other things to complete; as soon as it has to make an external call to get state, Durable Functions shuts down the orchestrator and you don’t pay for it anymore — so you only pay for the orchestrator when it’s not waiting around for external state.”

Durable Functions is obviously a good match for Azure services like Event Grid and Event Hubs, but it works well with third-party events too. Anderson mentioned using Twilio and a webhook request to Functions to add 2FA to Slack without needing to pull out a phone.

“Under the covers, it’s an HTTP request. You can have another function that acts as the front end for that external event, or the external event can call the admin endpoint directly,” Anderson said. “In practice, you generally have a function sitting in front and you can then have a couple of different triggers, like a queue or an admin panel with an HTTP endpoint and they can both trigger the function depending on which workflow makes sense.”

Azure Logic Apps workflows have been the main option for orchestration so far (and Durable Functions can run alongside Logic Apps), but that hasn’t suited developers who prefer to write code than use a graphical interface.

Currently, workflows can only be composed in C#, but it will cover other languages. Plans are afoot to extend the service to accommodate code written in Node.js, JavaScript, and Typescript.

Although Azure Functions has a persistent file system behind the functions, rather than the usual read-only copy (because it was built upon Azure App Service and WebJobs), but that’s not a good place to store state because it’s so expensive. Instead, Durable Functions uses Azure storage queues and Azure table storage for key-value pairs. “Eventually we’ll give you options of using other underlying Azure technologies for higher performance,” Anderson said.

“We hope to have additional providers that let you move onto using Azure Service Bus queues and Cosmos DB for the higher throughput scenarios. Usually, you have to use something like Kafka or Event Hubs to manage that kind of throughput,” Anderson said.

Theoretically, that abstraction could make it possible for Durable Functions to work locally (as the Functions runtime already can). Whether that happens will depend on customer feedback, but Anderson talked hypothetically about being able to use Redis, for example, as the storage for Durable Functions.

Currently in preview, Durable Functions is likely to move to general availability with version 2 of Azure Functions (which it depends on). As well as the planned expansion of storage options and broader language support, it will add more scaling options in time. “We plan on doing auto-sharding for you, so you can take advantage of many instances and get parallel scale through that.”

But rather than expanding the scope of Durable Functions, Anderson hopes to see more frameworks and projects aimed at making it easier to do large-scale development with serverless.

“We had to make this tradeoff [with serverless]: you don’t have to manage infrastructure, but you do have to give up a lot more control over your application pipeline, and there are not many frameworks — if any — that work when limited down to a single function entry point. My expectation is that we’ll see a new generation of frameworks that operate at the single functions level,” Anderson said.

Microsoft is a sponsor of The New Stack.

Feature image via PXhere.

The post Azure Durable Functions: Making Serverless Last appeared first on The New Stack.

Thanks to techniques such as deep learning, speech recognition keeps getting more accurate; depending on who you ask and how you measure it, deep learning based speech recognition systems might be better than human transcription now. Developers can call APIs that bring speech recognition to their own apps and services, but is it as good as the research results make it sound?

In 1996 the error rate for speech recognition was over 43 percent. By 2001, that was down to about 20 percent; outside of limited vocabularies like numbers, out of a sentence of ten words, two of them would be wrong. But unless users were prepared to invest a few months training recognition systems, the error rate stayed much the same for the next decade, plateauing around 15 percent. The hand-crafted generative speech models for recognition weren’t producing any more significant improvements and researchers from Geoffrey Hinton’s team at the University of Toronto worked first with Microsoft Research and then Google and IBM on using deep feedforward neural networks instead and error rates started dropping again.

When research scientist Francoise Beaufays started working at Google in 2005, speech recognition was still treated as science fiction, she told the New Stack. “We started by building small products, like a 411 service completely powered by voice recognition.” They were also using the older, statistical speech recognition models. “Part of the improvement was gaining access to more data and training with it, but it was also about evolving the technology,” she explained. The biggest breakthrough in this period was moving to neural networks while keeping the latency low enough to give results quickly.

In speech recognition, there are three models that work together: the acoustic model, the pronunciation model, and the language model. The acoustic model takes the waveform of speech and chops it up into small fragments and figures out each sound that the person is speaking. The pronunciation model that takes those sounds and strings them together to make words, and a language model that takes the words and strings them together to make sentences. These models are computed together into one big storage graph.

“When you speak a sentence. the sentence is pushed through the graph and we try and find the path of highest accuracy; the sequence of words that we think with the highest confidence is what you meant to say,” she said.

Moving to neural networks didn’t necessarily improve recognition accuracy immediately; “what it created was a path for a bunch of innovations,” she explained. “After the initial launch, every six to nine months we come up with a new architecture, a new type of neural network that’s more powerful, more efficient, less [susceptible to] noise.” The same continual development has improved error rates across the industry, although measuring that turns out to be tricky.

Common Benchmarks

The standard test for measuring the accuracy of speech recognition uses a fairly old dataset called NIST 2000 Switchboard: recordings of telephone conversations between strangers, about common topics like sports and politics. “This is the standard speech set used by all researchers for the last 20 or so years,” Microsoft Technical Fellow Xuedong Huang explained to the New Stack. “It’s natural conversations that people had when they talked over the phone; they’re not talking to their own family, so they’re using standard English.”

In 2016, a Microsoft Research team got the error rate on the Switchboard corpus down to 5.9 percent. “That matched the human error rate when we hired a professional transcriber to transcribe the same data,” Huang told us.

Since then, there’s been some debate about how accurate you can get a human transcription to be. IBM used four teams of people, doing multiple transcriptions of each sentence and combining their work got the error rate as low as 5.1 percent (and reported an error rate of 5.5 percent using IBM Watson ). “When you have a group of people working together, the new error rate for humans is a historical low of 5.1 percent, and we just reported that with software we can match a group of people.”

Huang puts the improvement down to the speed of the latest version of the Microsoft Cognitive Toolkit (CNTK) for deep learning, which lets the team find more effective models more quickly. “It’s amazingly fast, and we were able to run experiments two to three times faster than we used to. That improved our productivity; we experimented with a thousand different models to get the final set of models that got us to 5.1 percent.”

But what does matching human performance on Switchboard mean for the speech recognition systems users and developers get access to? “This is a good milestone, but it by no means implies we’ve solved all the problems,” he pointed out. The advantage of benchmarking with Switchboard is an “apples to apples” comparison.

Beyond the Switchboard

Yet, Google doesn’t benchmark its speech recognition against Benchmark because “those tests are not relevant to our product and our users,” Beaufays said. “We have our own test set and we try to beat that, but comparing across companies is difficult. The audio is not representative of how people are talking; that material is not relevant to us. We have our own product and we optimize for that.”

Google product manager Dan Aharon quoted the 4.9 percent error rate that was announced at Google I/O this year and noted that as Google has made speech recognition available to customers as an API, it’s had to be adapted for customers who didn’t have the same needs as Google.

“It became obvious pretty quickly that large segments of customers were interested in using speech for things different from what Google has traditionally done and our initial model was not well suited for those kinds of use cases,” Aharon noted. “Over the last year or so we’ve worked to improve that and we’ve made a significant R&D effort to get better technology tuned for phone call transcription, for transcribing long-form audio and for multiple speakers.”

Both Microsoft and Google let customers add their own vocabulary, like product names, to improve accuracy; Microsoft also supports custom audio models for specific locations or age groups. Because YouTube has products aimed at younger users the team trained a system that was adapted to their voices.

“They not only have a different speech range but often choppy ways of expressing themselves that could force a different interpretation of the words if we didn’t pay attention to that,” Aharon explained. But rather than supporting custom models, “we found merging that with the overall system gave us a speech recognition system that works well for young and adult people.” (That’s used for Google Home but for legal reasons, the Google Cloud speech API doesn’t support apps directed at children under 13.)

Similarly, Microsoft’s 5.1 percent error rate came from training the system on more speech data than just Switchboard, or the Call Home project which collected and transcribed voice recordings in the early 1990s by offering US college students free long-distance calls. That’s more like colloquial everyday speech, but it’s still more than 20 years old. Google has been working with machine learning training company Appen (which also helped IBM with its group transcription test) to collect recordings from people with particularly challenging accents, covering Scottish dialects and the rural Midwest, and Microsoft has its own data sets too.

More than 10 percent of search is now made by voice and not long ago it was less than 1 percent.

Huang didn’t put figures on the accuracy of the Cognitive Services speech API, the new Harmon Kardon Invoke speaker that uses Cortana speech recognition, or the PowerPoint add-in that both transcribes and translates your presentation, other than saying it would be a higher error rate. “We share the same training tool between CNTK and the production system, and we use more data than for Switchboard benchmarking, but this is on demand and real-time; Switchboard is not. Also, far field is more challenging than a close-talk microphone because the signal to noise ratio is worse.”

Even though the recordings were made from older phones, the Switchboard recordings were done with handsets that put the microphone close to the mouth. To recognize speech across the room, speakers like the Invoke use multiple microphones; Google Home has two, Amazon Echo seven. “Noise is a challenge: is it manageable? Yes, if you have the right microphone. This a beamforming microphone array and we make sense of multiple beam forms to get the right person and we use signal processing to enhance the source of speech and make the overall accuracy better,” Huang explained. “With a beamforming array microphone, it’s close to near-field accuracy. With a PC open microphone, it’s not as good.”

The PowerPoint system needs a close-talk microphone; “this isn’t 5.1% error rate because it’s real-time,” he noted, “but it’s intelligent; it can learn from the PowerPoint content and if you talk about what’s in the slide, it can be further enhanced. It’s not the most sophisticated neural network, it’s not as good as the 5.1% research system — but it’s still highly usable.”

The Real Test Is Using It

Beaufays (who has a noticeable French accent) also suggested that the real test is whether people find speech recognition useful. “It’s really hard to assess the quality of a speech recognition system. In a real context, it’s really hard and quite subjective. If you have a person speaking to the phone and it’s inaudible because they have the music on full volume, expecting speech recognition to get all of that is a grey area.”

He noted that reaction from users is reflected in the adoption numbers: more than 10 percent of search is now made by voice and not long ago it was less than 1 percent. Instead of typing on their phone, users are taking that leap of faith.

And Huang highlighted challenges beyond just the error rate. “When we communicate as people, the performance drop from close talk near field to far field is not as dramatic as with machines. Machines do very well almost as good as humans if it’s close talk. If you have a little bit of an accent the tolerance level of machines today is not as good as humans; for a three-year-old kid, it’s going to be disastrous. If you have very far field conversations, people have this amazing ability to zoom in, to focus their attention and ignore everything else.” That’s called the cocktail party effect, and Mitsubishi Electric recently demonstrated a system that can distinguish two or three voices in a crowd.

That’s more a symptom of a bigger underlying issue, Huang suggested. “Most important of all is the understanding; humans use intelligent contextual information like body language and eye contact to really enhance our understanding [of speech]. People aren’t trained to transcribe every single word; even with grammatical mistakes and if the sentence isn’t perfect, we understand it based on the context. That’s the biggest gap where machines are far from getting close to human capability.”

Google and Microsoft are sponsors of The New Stack.

The post Beyond the Switchboard: The Current State of the Art in Speech Recognition appeared first on The New Stack.

Microsoft has expanded its Azure-hosted Cosmos DB to serve as a drop-in replacement for Apache Cassandra data store.

The company Microsoft launched Cosmos DB earlier this year as a hosted JSON document database, though the architecture was always designed to support multiple APIs and data models, giving it the ability to mimic the interfaces of other data stores. Now Cassandra is joining the MongoDB, Gremlin, standard SQL, and Azure Tables as database systems Cosmos DB supports natively.

The company announced the update during its Connect() 2017 event, held Wednesday in New York.

The new API support, released in preview today, gives developers “Cassandra-as-a-service powered by Cosmos DB,“ said Rimma Nehme, Microsoft group product manager for Cosmos DB and Azure HDInsight. “You get the familiarity of your favorite Cassandra IDEs and tools, and you can take your existing apps and make them planet-scale by changing the URI and pointing to our platform.”

Support for Cassandra has been one of the top requests from customers, she told The New Stack. Some of them want the simplicity of a managed service, with the security and compliance of a cloud service. “Cassandra developers frequently build large-scale distributed system and that comes with a lot of burden and complexity. By having this API, out of the box, Cassandra developers get a fully managed Cassandra as a service, as a true PaaS service that takes advantage of Azure.”

Others have built their own Cassandra systems that they need to grow beyond the resources of their own data centers — but don’t want to rewrite the application they’ve already created. “Typically, these are the customers who are pained by the scale issues, whether they have large-scale data sets or they particularly need the global distribution,” she said.

One of the big advantages of Cosmos DB is that it offers more than the usual range of consistency models, with intermediate consistency models of bounded staleness, session consistency and consistent prefix, which have clear trade-offs. “It’s a fundamental choice between performance, latency and availability, but also the dollar costs. The stronger the consistency that you have, the more processing you have to run, the more distributed algorithms, it consumes more of your throughput, so you will have to pay more for it,” Nehme said.

More than 93 percent of Cosmos DB customers pick one of the intermediate consistency models, often the less expensive session consistency, she noted.

Compatibility Commitments

Cosmos DB offers what Nehme called “wire protocol level compatibility” with existing Cassandra SDKs and tools. As a result, developers may not have to rewrite their apps when switching the backend to Cosmos DB.

This isn’t the using existing open source Cassandra database code; instead, Microsoft has implemented the API and wire protocols as modules on top of the Cosmos DB atom-record-sequence (ARS) data model. In the preview, not all of the Cassandra APIs are included; Nehme said most of them are covered but the support isn’t yet as extensive as the MongoDB coverage.

“The typical patterns of Cassandra usage are actually more of the CRUD [create, read, update and delete] workloads than queries. All of the CRUD functionality is working, the querying capabilities are working. We’ve taken a few canonical tier one workloads on Cassandra and made sure that we’ve got everything covered and we will monitor customer workloads and address the last mile things to take it to general availability.”

The support is good enough that some early customers for the private preview are already running workloads in production. Jet.com is already running what Nehme called “several mission-critical use cases” with Cassandra APIs on Cosmos DB.

Nehme expects some customers to continue using Cassandra in their own data centers, alongside running the same database applications on Cosmos DB, which has happened with MongoDB developers. Sometimes that’s as part of a phased migration to the cloud; “you might not cut the umbilical cord overnight, you might have it running side by side in parallel for some time.” Others will keep the open source NoSQL databases as local development tools.

Multiple Models – At Once

With so many APIs supported now, Cosmos DB gives developers the option of using multiple data models and query APIs on the same data. “If you look at the apps being built in the cloud a pattern we see frequently is polyglot apps with a very fragmented backend,”  Nehme said. One database may run the transactional workload, while a key-value store may capture the Internet of Things telemetry data. a document database stores inventory or user profile management and a graph database are used for connections between customers and suppliers and other parties involved in the business.

“By having one database that gives you this multi-model support, that’s applicable to all of those data models seamlessly,” she said.

Even if no code needs to be rewritten, moving to Cosmos DB does mean translating Cassandra settings. To help with migration, Microsoft is working on documentation that maps settings Cassandra developers will be familiar with to the five consistency models available in Cosmos DB. It’s also planning to help all developers choose the right consistency model for their workloads by adding a graph to their database metrics showing how likely your data is to become consistent, using the Probabilistically Bounded Staleness model, which attempts to answer the question of “how eventual is eventual consistency?”

“If you’re building an app using Cosmos DB for web comments or more exploratory workloads, it’s OK if your data is not perfectly consistent; if you’re building your shopping cart, you want to make sure the data is reflected in a strongly consistent manner right away,” Nehme explained.

Beyond Cassandra support, Cosmos DB has been augmented in other ways as well. Cosmos DB’s MongoDB support is also improving with a new public preview of aggregation framework support and unique indices. “Unique index allows you to introduce uniqueness constraints on any document fields on already automatically indexed documents, which is a core capability of Cosmos DB, and aggregation pipeline allows you to run more sophisticated aggregations on top of your data using MongoDB native APIs.”

The Table API is moving from preview to general availability, and the Gremlin API will soon follow, with improvements to performance, import and backup, plus Python client support and more options for using the open source frameworks recommended by Apache Tinkerpop. Nehme also promised ways to simplify migrations from Neo4J and Datastax’s TitanDB. With TitanDB, Cosmos DB could serve as a replacement instead of  JanusGraph, a fork of the graph database that took place after its acquisition by Datastax.

The next API to be added to Cosmos DB will likely be Hbase, which fits in with the existing Spark connector. That might even remove some of the artificial divisions between operational and analytics databases introduced when database systems couldn’t cope with running both at the same time.

Microsoft is a sponsor of The New Stack.

Feature image: Microsoft technical Architect Lara Rubbelkey, introducing Cosmos DB improvements, at Microsoft’s Connect 2017 conference in New York.

The post Microsoft’s Cosmos DB Gets Cassandra-Compatible APIs appeared first on The New Stack.

Microsoft’s new Azure Kubernetes Service (AKS), now in preview, extends the company’s existing Azure Container Service into a more managed environment, one built around the Cloud Native Computing Foundation’s Kubernetes container orchestration engine.

“We heard very clearly from customers that what they want going forward is a fully managed experience,” said Microsoft’s lead program manager for containers on Azure, Gabe Monroy. “The biggest difference is that [Microsoft’s basic Azure Container Service] is offering an unmanaged experience whereas AKS is a managed experience.”

Basing that managed service on Kubernetes isn’t Microsoft picking a winner the way it would have in the old days; nowadays the Microsoft strategy is to go where its customers are and to support a full range of options. That’s why the Azure Container Service supports Swarm and DC/OS as well as Kubernetes; there’s still a lot of demand for those and these alternate orchestrators will continue to be “first-class solutions on Azure” he confirmed. The container service isn’t going away, but AKS is the next version of the service.

“I think we’ve seen both inside Microsoft as well as outside, a pretty rapid consolidation around Kubernetes as sort of a de facto open source container orchestrator,” Monroy told us.  The strength of the Kubernetes community and ecosystem has a lot to do with that.

As Kubernetes co-founder and Microsoft distinguished engineer Brendan Burns told the New Stack recently, orchestrators should be boring. “The specific difference in features are just not very important for customers,” Monroy pointed out. “They’re really looking to move on from that layer and focus on the next set of problems like service mesh technology, developer tooling, security and how to integrate all of this into a coherent software lifecycle and governance framework. Kubernetes out of all the orchestration options has the strongest ecosystem, so if you’re trying to solve a lot of problems, doing that with Kubernetes is going to be easier.”

Even large organizations in traditionally conservative and regulated areas like financial services are adopting Kubernetes containers at what he called “breathtaking speed.” “The value proposition of containers and orchestration are sufficiently promising that folks are ready to make bets on them despite the lack of managed services being widely available.”

Containers and Beyond

As a new service still in preview, AKS doesn’t have all the features that are in the basic container service, although Monroy told us the gap will close rapidly, because “we really see AKS as the future of our next service offering.”

In preview, only Linux containers are supported — because some core Kubernetes components still have to run on Linux hosts — but both Windows only and mixed Linux and Windows node pools are on the roadmap. Some of that depends on work being done in Kubernetes itself; there’s some progress in 1.9 and more improvements in 1.10, and the AKS team is targeting the first half of 2018 for Windows container support.

“Most of the features we’re being asked for right now fall into networking and security customization,” Monroy said, adding that they’re things competing solutions don’t offer, meaning “customers are asking us to lead the industry.”

The top request from users is support for integrating Kubernetes into an existing virtual network service endpoints (VNET), the way you already can with the basic container service. Custom virtual networks should be added before the service moves to general availability. Role-based access controls and network policies aren’t in the preview yet, but they’re planned for the release version.

Persistent volumes work the way they do in the container service; currently, Azure files, Azure Data Disks and Azure Managed Disks are supported. You can pick specific VM types for nodes as long as they’re available in the Azure region, although some VM types aren’t an option yet — adding Azure NC6 VMs with NVidia drivers is coming, and you can’t yet pre-attach managed disks to agent VMs. Auto-scaling is also planned, assuming the upstream Kubernetes auto-scaler is merged into the core release.

In preview, upgrading the control plane causes a brief (10-15 second) downtime window; both that and node upgrade will work without any downtime when the service comes out of preview.

The Kubernetes dashboard is available with AKS. Defaults can’t yet be changed but there will be options for changing some of them before launch. PowerShell support will also come by the time the service reaches general availability, and all the AKS functions will eventually be supported through the Azure CLI, the Azure portal and the Azure APIs as well.

Because the basic container service will still be available there’s no automatic migration to AKS, but Microsoft is working with Kubernetes service provider Heptio to make sure tools like Heptio’s Ark will support migration to AKS, including options like persistent volumes.

The preview was initially available in the East US and West Europe regions, followed by Central US; West US and UK West seem to be the next regions coming, based on the region list on GitHub.

The initial demand was much higher than Microsoft had expected. “We were really surprised by how quickly people dove head first into some of these deep technical issues,” Monroy said. “Typically, with this sort of infrastructure software when you release it people don’t immediately have time to kick the tires; that requires time and the type of operational person doing these evaluations has to wait until their schedule frees up.”

Although he wouldn’t commit to Microsoft bringing AKS to the company’s on-premises version of Azure, Azure Stack, he did confirm that the team is discussing the possibility and is “excited about the prospect.”

Integrating with AKS

AKS isn’t an Azure-specific flavor of Kubernetes, but it will get deep integration with Azure services to bring those to the customers who are picking Kubernetes for its portability and openness by exposing them through the Kubernetes APIs.

“The goal is for it to be solid and reliable and a plumbing service that is part of the infrastructure. We don’t want to turn AKS into a full-blown platform; the idea is to provide a Kubernetes API as a service. However, we do want to make it a great platform on which to build, so we’re working with a lot of different partners to take this piece of infrastructure plumbing that we’re building and really flesh it out to a much bigger solution.”

The Cloud Native Computing Foundation and Microsoft are sponsors of The New Stack.

Feature image by Jason Blackeye via Unsplash.

The post Microsoft Centers Azure’s Container Services Around Kubernetes appeared first on The New Stack.

Running a virtual machine (VM) inside a VM might sound inefficient, but hardware-assisted virtualization on recent CPUs means there’s only a moderate performance penalty and this nested virtualization unlocks a lot of options for traditional applications and for applications that mix and match containers and VMs.

Microsoft Azure has the feature live already (it’s part of the Windows Server 2016 feature set that Azure runs on, and it’s available for recent VM instance types in all regions). Google Compute Engine recently unveiled a beta version for KVM and Linux VMs running on instances with Haswell or newer CPUs, and one of the reasons for Amazon Web Services upcoming switch from Xen to KVM for its C5 instances may well be to enable nested virtualization (especially since Amazon contributed some related code to the Linux kernel earlier this year).

The idea isn’t new; when Microsoft bought Virtual PC from Connectix in 2003, it could run its own emulator. But it was Hyper-V Containers — which combine the advantages of Docker containers with the isolation of VMs — that gave the Windows Server the impetus to deliver a feature they’d wanted to create for a decade.

“What we found it’s this very handy, helpful tool, especially for developers, but the use cases for nested virtualization explode very quickly,” Microsoft’s Taylor Brown told The New Stack.

Lift, Shift and Append

“Our primary stack on Azure is all virtualized,” Corey Sanders, Microsoft’s head of product for Azure Compute pointed out. “Some customers have the need to take advantage of some virtualization technologies and capabilities and our host won’t expose those directly.”

The most obvious uses are for QA and for running training labs, where you want to move an existing, virtualized environment to the cloud, without making any changes, and those are the same examples Google gives for nested virtualization, with services like Functionize.

But the same is true of production workloads, Brown said.

“I can move this entire environment into Azure that I couldn’t do before. I’ve got a set of VMs that are preconfigured and I’ve validated them and they work; now I don’t have to worry about changing anything, I just lift the entire image and run it in Azure and all of those VMs will keep working the way they did before. Now I can move entire applications that can’t easily be containerized or lifted and shifted; that whole piece of this application just runs in a set of VMs, nested, even it’s one to one.”

Unlike using, say, Oracle’s Ravello nested virtualization to run VMware workloads on AWS without changes, there’s also an opportunity to extend the app and replace one-off “snowflake” environments by codifying them.

Nested virtualization also gives third-party software vendors a way to package up existing applications and services for the cloud. “We have a partner who had built a solution around Hyper-V Replica, which is an API that enables you to take a snapshot and do live replication of Hyper-V to another Hyper-V instance,” Sanders confirmed. “They want to enable that solution in Azure, and with nested virtualization, they can have their customers deploy a VM inside our VM and take advantage of that technology that’s built into Hyper-V.”

Some customers have been asking for nested virtualization to give them more control over their VMs on Azure. “With nested virtualization, you could create whatever instance sizes you want inside our size and even configure oversubscription as part of that to try and utilize cloud hardware in more efficient ways.” That requires what Sanders tactfully calls “a lot of sophistication” and also sacrifices the simplicity and agility of cloud IaaS.

Containers with the Isolation of VMs but not the Overhead

But the biggest interest from Azure customers is in the way nested virtualization can combine the lower costs and easier servicing model of containers with the isolation and security benefits of VMs, using Hyper-V containers. “The big picture with nested virtualization is going to be containers,” Sanders predicted.

“Especially for those customers who are running some customer-facing apps, some multi-tenant solutions where their end customers have full access to containers, enabling that layer of virtualization protection and isolation is going to be important,” he said.

Not everyone needs that level of protection, but it’s what makes containers viable for those who do, Brown explained. “We’re seeing enough penetration with Hyper-V isolation that it’s clear there’s a business and regulatory need for a well-established isolation boundary.”

“Hyper-V isolation allows us to have a conversation where we say ‘the security boundaries are no different, they are exactly the same as with physical’, but containers are a different way of thinking about a problem space today. The property that VMs offer is they’re heavily safe which is valuable for some apps but really unnecessary for others. So, we can have a conversation about ‘why do you need a stateful tier or not a stateful tier? why does that particular tier need that state?’ We can have a conversation about things for developers like checkpoints; VMs offer checkpointing technology which captures all your memory state. Is that what you need, or do you just need the on-disk state; if all you need is the on-disk state, then would containers be a better fit for you?”

This development mirrors both the shift to containers and their continued evolution, Brown suggested. “VMs were doing fine. They solved a lot of our needs in the cloud, but containers offered a better experience for portability across clouds and the DevOps model. They were a step above that level of abstraction but they still provided much of the same value and that’s part of why they’ve been so attractive.”

As developers started finding gaps in what they could do with containers, new tools were needed. “As we find areas where the answer is ‘we need this other tool for the job’, like statefulness — which is to separate the state — [you get] things like volume drivers and the ability to persist state outside of the container in a very well understood way. So we get the better density offered by containers and the DevOps model but we can still use it for some stateful applications.” Running containers in nested virtualization fills another gap.

Real World Solutions for Specific Problems

Nested virtualization isn’t how Microsoft is providing its new VMware support though (nor the SAP HANA on Azure support it recently announced). “There’s a fair set of capabilities that we have announced on top of bare metal, like SAP large instances and Cray,” Sanders explained. “In all those cases the support on bare metal is tied to being only able to do it on bare metal for a variety of reasons, whether it be network requirements or scale requirements or actual hardware requirements as with Cray. In the VMware case a lot of it was around networking requirements that were necessary to deliver it.”

Nested virtualization doesn’t support broadcast or multicast, for example, and Brown told us the Hyper-V team wants to do more work “improving the memory access and performance.”

Hyper-V containers using nested virtualization on Azure and Windows Server initially supported just Windows containers, but Microsoft has been working on running Linux containers as well (abbreviated as LCOW for Linux Containers on Windows). Windows Server 1709 can run Linux containers with Hyper-V isolation, although the Docker LinuxKit support is still in preview, step-by-step instructions for Ubuntu are here; create the container in a VM with a vCPU, like a Dv3 or Ev3 instance, and it runs with nested virtualization.

The next stage, which is still being merged into Docker, is support for running Linux and Windows containers side-by-side on the same Windows node at the same time (not just mixing Windows and Linux nodes in the same cluster, which you can already do). That will give better latency between the containers, but it also allows organizations to run mixed loads on a single infrastructure and makes it easier for developers to build mixed Linux and Windows applications on a single system. And it gives businesses in regulated industries who want VM-level isolation for Linux containers another option than traditional virtualization.

So far, AWS and GCP only support KVM, and only with Linux. Although Sanders noted that Hyper-V is “the hypervisor we fully understand and fully support,” Azure’s nested virtualization works for other hypervisors as well; Azure’s own networking team uses nested virtualization with KVM to run containerized router and switch images alongside those available as VMs in its CrystalNet network emulation tool — because some network hardware manufacturers only offer images as VMs and some only offer them as containers.

That kind of mix and match, heterogeneous system might well become more common as organizations go beyond migrating applications to the cloud and start extending and enhancing those applications. It’s rare to be able to start from scratch; often the resources an application needs won’t all be available in the same clean, neat, modern form.

“Almost every application in the world, if it was completely rewritten today, would use much different technologies but for almost every customer and application that’s just not a rational or reasonable possibility,” Sanders said.

Google, Microsoft, and VMware are sponsors of The New Stack.

Feature image by Kate Remmer on Unsplash.

The post Nested Virtualization Offers Containers the Isolation of VMs without the Overhead appeared first on The New Stack.

Ranking isn’t just for search engines, or even enterprise search, although it’s routinely used by services like Airbnb, Etsy, Expedia, LinkedIn, Salesforce and Trulia to improve search results.

The Learning To Rank (LETOR or LTR) machine learning algorithms — pioneered first by Yahoo and then Microsoft Research for Bing — are proving useful for work such as machine translation and digital image forensics, computational biology, and selective breeding in genetics — anything you need is a ranked list of items. Ranking also are quickly becoming a cornerstone of digital work.

Beyond document retrieval, LTR is useful for expert search, definition search, collaborative filtering, question answering, keyphrase extraction, document summarization, expert finding, anti-web spam, sentiment analysis, and product ratings. LTR methods work for images as well as text (Etsy uses a combination of image and text search to distinguish between products that would get a very similar ranking based on the text description alone, and LTR can distinguish higher and lower quality images), or for ranking nodes in a graph. And they’re increasingly available to developers, whether you’re looking for a Python library or a full-scale enterprise search platform like Apache Solr/Lucene. (Lucene LinkedIn’s old search framework; it’s no longer used there but was open sourced and is used by sites like Pandora, Soundcloud and Wikipedia. Solr is the open source enterprise search platform built on Lucene.)

“LTR is suitable for pretty much any area where you can find unbiased training data,” Ganesh Venkataraman, engineering manager at Airbnb (and former head of relevance at LinkedIn) told the New Stack. “It is particularly suitable for cases where there are a large number of features that are harder to hand tune.”

“LinkedIn uses LTR on almost all of its search verticals — people search, job search, recruiter search and so on. We have shown this to work very well with sizeable performance gains and have also seen it scale,” Venkataraman said. For work such as the job search, the technique ties very closely with another algo, query understanding (QU), which predicts the user intent based on query. For example, if the job search query is “google software engineer,” LinkedIn will understand that the member is most likely looking for jobs at the COMPANY “google” with the TITLE “software engineer.”

Using LTR for matching search queries with relevant ads on Microsoft Bing improved online ad revenue by $4 million in the first two months. Using it to rank job adverts on LinkedIn didn’t just get more people clicking through to look at search results; the search was finding jobs people were looking for, because application rates went up 5 percent in the first three weeks (and up 50 percent by the end of the project). When Bloomberg built an LTR plugin for Solr, not only was it a third faster than the hand-built ranking system they’d been using before, results were more relevant, and clicks went up by about 10 percent straight away.

How LTR Works

Learning To Rank uses supervised machine learning to train a model not for the usual single-item classification or prediction, but to discover the best order for a list of items, using features extracted from each item to give it a ranking. It’s not looking at the precise score for each item but the relative order – whether one item is ranks above or below another.

For web results, features might include the domain or PageRank, whether the page title matches the query, the age or popularity of the page, whether it has standard document sections like an introduction or a summary, how long it is and how long people spend reading it, or even whether it will render on a phone screen or just a desktop browser. To make ranking efficient, features need to be extracted automatically; they should be something you can script rather than something you have to code by hand.

The three main LTR techniques are pointwise, pairwise and listwise. Pointwise LTR uses regression or classification to discover the best ranking for individual results. You can think of classification as putting similar documents in the same class and regression as giving similar documents a similar function value, so ranking assigns similar documents similar preferences.

Pairwise LTR uses classification or regression to discover the best order for a pair of items at a time, classifying all the different pairings for items in the list as correctly or incorrectly ranked and working through them to get the ranking for the whole group. There are multiple methods using different techniques for both the cost function and the learning algorithm, including neural networks, random forests, boosting and Support Vector Machines (SVM).

Early pairwise methods like RankNet used a cost function that’s about minimizing the number of times a lower rated item is ranked above one with a higher rating and Stochastic Gradient Descent to train the neural network. LambdaRank gets faster, more accurate results using the gradient of that cost (whether an item should move up or down because of its ranking, and by how much), and LambdaMART improves on that using gradient boosted decision trees.

Listwise LTR methods like ListNet rank the whole list rather than working through pairs, using probability models for the cost. For images, using a subset of the list of images can work better and combining all three cost functions can also improve ranking performance.

The first search engine to use LTR was AltaVista (which was bought by Yahoo) and the key work on LTR was done between 2008 and 2011, with Yahoo and Microsoft releasing training data sets and holding competitions, and the techniques from that era are the ones that are commonly used — but there’s still plenty of on-going research. Venkataraman called “the intersection of deep learning and IR (information retrieval)” particularly important; recently, he says, “Word embeddings (converting query tokens into a higher dimensional space), as well as neural nets, got quite some attention.”

There’s also research into semi-supervised versions of LTR. Supervised learning relies on well-labeled training data from human experts (like the human judges Google uses to get test scores for changes to its search algorithm), which offers high-quality data — but it’s expensive and only covers a fraction of whatever it is that you want to rank. Analysing search logs and click-through rates is cheaper and works at a much larger scale, but there’s often bias in that kind of implicit data (the top results will get more clicks but may not be the best match, and if none of the results are clicked you get no useful information).

Tools for LTR

Several LTR tools that were submitted to LTR challenges run by Yahoo, Microsoft and Yandex are available as open source and the Dlib C++ machine learning library includes a tool for training a Ranking SVM. The pyltr library is a Python LTR toolkit with ranking models, evaluation metrics and some handy data tools.

For most developers, LTR tools in search tools and services will be more useful. LinkedIn open sourced sample code for building an end-to-end ‘instant’ search system on Elasticsearch that uses LTR. The Azure Search service for building custom search engines has LTR functionality built in for relevance ranking; the custom code Ernst and Young built to create a custom search engine for the US tax code is on available as Python scripts and Jupyter notebooks. That’s a good example of combining LTR with natural language processing (in this case using Microsoft’s LUIS Language Understanding Service API) for extracting features for ranking documents.

Bloomberg’s LTR plugin is now built into Solr; it extends the existing RankQuery in the platform and reranks a number of documents using your ranking model (which is deployed as a Solr managed resource) and feature information (so features can be built in Solr with short, simple scripts and rich search options like synonyms and phrase matching still work). That makes sophisticated results ranking available relatively simply, in a popular search platform.

Microsoft is a sponsor of The New Stack.

Feature image by Mar Newhall on Unsplash.

The post Learning to Rank: A Key Information Retrieval Tool for Machine Learning Search appeared first on The New Stack.

The Microsoft PowerShell task automation and configuration management framework has long been associated with core Microsoft workloads; it provides access to everything from Azure Stack to Windows 10. But since mid-2016, it’s also been a cross-platform open source shell and scripting tool, one uniquely suited to the emerging world of APIs and structured objects.

Increasingly, Microsoft’s customers don’t just use Windows and Windows Server; they use Linux and Macs (plus Raspberry Pi at one end of the scale and multiple cloud services at the other). Supporting this diversity, PowerShell Core runs on Windows 7, 8.1, and 10, Windows Server 2008 R2, 2012 R2, 2016, Windows Server Semi-Annual Channel, Ubuntu 14.04, 16.04, and 17.04, Debian 8.7+, and 9, CentOS 7, Red Hat Enterprise Linux 7, OpenSUSE 42.2, Fedora 25, 26 and macOS 10.12+, with unsupported packages for AppImage, Arch and Kali Linux, and experimental versions for Raspbian and Windows on ARM.

“More and more of the Linux world is becoming API based and returning structured objects, JSON objects which we convert to text and can do all sorts of wonderful things on without any parsing.” — Jeffrey Snover

The Linux version of PowerShell is no longer a separate code base; along with macOS and what looks like an increasing number of platforms including IoT, it’s part of the main PowerShell project, with common features and functionality. “The company is becoming a cross-OS company. I like to say that ‘the sea refuses no river’; we want to be the company for everybody, no matter what platform you’re using,” said Microsoft Fellow Jeffrey Snover.

But the general availability of PowerShell Core 6 for Windows, macOS and Linux doesn’t just mean Microsoft’s solution for managing the heterogeneous mix of systems now common in business is ready to use in production. It also brings some major changes to PowerShell on Windows, because the cross-platform, open source, community-driven, “get-it-from-GitHub” Core version of PowerShell is where all new development will happen.

Truly Cross-Platform

What we’ve been calling PowerShell for the last decade is now named Windows PowerShell, because it’s built on the .NET Framework that’s only available on Windows. PowerShell Core builds on Microsoft’s (also open source and cross-platform) .NET strategy, using the .NET Core 2.0 runtime and the .NET Standard 2.0 common libraries and APIs.

“What we’re doing with PowerShell Core is we’re hitting the refresh button,” Snover told us. “We’re keeping all the things that were great and we have the courage to leave behind the things that don’t bring us to where we want to be. And where we want to be is a tool that allows companies to manage anything from anywhere.”

That means little things like PowerShell Core now handling common Linux filename characters like colons and commas that Windows doesn’t support, as well as paths with forward or backslash separators. The default encoding is UTF-8 without a Byte Order Mark; the default on Linux and macOS and increasingly common on Windows.

Logging support was added for GA, but it uses a different set of group policy objects than the Windows PowerShell settings (which will simplify using both side by side, but is another change to make once the .ADMX list is published). On macOS, PowerShell uses the native os_log APIs to log to Apple’s unified logging system; on Linux, it uses Syslog. For remoting, PowerShell Core uses SSH as a native transport protocol. Register PowerShell with an OpenSSH-based SSH server and existing SSH-based authenticate mechanisms like passwords or private keys work with the traditional PSSession semantics.

PowerShell Core isn’t attempting to replace Bash, Python or any existing tools; what it’s doing is bringing the automation and handling of structured data to Linux, at a time when that is becoming increasingly important.

“More and more of the Linux world is becoming API based and returning structured objects, JSON objects which we convert to text and can do all sorts of wonderful things on without any parsing,” Snover pointed out, claiming that “Linux is moving to a place where PowerShell’s sweet spot is. On Linux, we’re a tool, we’re another shell just like any other shell and there’s a gazillion shells out there. If your world is just text parsing we do that fine but we’ve no big advantage over some of the others. But if you have structured data, there’s where we pull ahead by quite a huge distance.”

PowerShell also offers Linux some advantages in security. “We now have the world’s first security transparent shell running on Linux. We took as much of the security stuff [from Windows PowerShell] as we could run on Linux, so you can configure things to have full transparent script block logging and so on, and Linux has never had that before,” Snover said.

PowerShell program manager Joey Aiello confirmed that structured data parsing is popular with Linux PowerShell users, as is cloud orchestration.

“In general, I think people are using a lot of cloud orchestration modules; we have a Core version of AzureRM and VMware and AWS have published some modules that people are using. But I also see very often PowerShell being used as part of test orchestration and general REST API automation. We have PSSwagger that makes it easier to generate PowerShell module from open API documents that describe REST APIs and that’s made it very easy to script against REST APIs, which we see as the next level of APIs to wrangle. We’ve moved up a layer in the stack; PowerShell has really done well as a funnel a vacuum for all these disparate APIs, and we think it has a lot of opportunity to do that again in in the cloud and REST API world.”

Side by Side

For Windows PowerShell users, the implications of PowerShell Core are slightly different, because they already have a tool they’re used to and have invested effort in learning, as well as this new tool which has slightly different features that isn’t installed automatically. In fact, it’s only in the last two months that Windows use of PowerShell Core has become significant; “before that the vast bulk of usage was Linux,” Snover told us (to his surprise).

PowerShell 5.1 can run side by side with PowerShell Core, which has its own pwsh.exe executable and comes as a ZIP package so admins can install multiple versions, and scripts can be pinned to specific versions that they require. Windows PowerShell it will continue to be shipped and supported as part of Windows 10 and Windows Server, and it will get bug fixes. But it won’t get new features, like the enhancements in the JSON and Web cmdlets to simplify writing scripts against web services, or the new Docker support that lets you use docker pull microsoft/powershell to get PowerShell Core.

“Windows PowerShell 5.1 is a very strong, very capable tool that continues to be supported and will be supported, my guess is forever, but it not going to be enhanced,” Snover told us. “I’m confident that anybody who needs to do something in a Windows-only way will be perfectly fine with Windows PowerShell. But we’re going to put all our investment in the new cross-platform tool.” In other words, Windows PowerShell isn’t going away, in much the same way that CMD.EXE has stayed in Windows for those who find it useful.

That’s still caused some concern among Windows PowerShell users, especially about compatibility for modules and tools, as well as differences in functionality. The goal is to be as compatible as possible. “In order to be cross-platform. there were quite a few little changes that we had to make to better fit into the UNIX world and the reality is those things were breaking changes, so there is a difference between Core PowerShell and Windows PowerShell,” Snover noted. “In reality, the vast majority of peoples’ things are just going to work when the existing cmdlets work on .NET core but there are some changes.”

If the .NET features that a PowerShell module depends on are in .NET Standard 2.0, that module will work with PowerShell Core, he explained. “The WindowsPSModulePath module changes your module path and brings in the existing modules and the reality is that most of those work just fine; some of them don’t but most of them do. More importantly, though, the team [that built that module] hasn’t signed up to support them, they haven’t done full testing.”

Most of the modules that ship as part of Windows (like Hyper-V, NetTCPIP and Storage), or with other Microsoft products like including Azure and Office have not been explicitly ported to PowerShell Core yet and aren’t supported but many seem to work. Many of the built-in modules that come with PowerShell itself have bug fixes and new capabilities — but sometimes they also lose functionality if .NET Core doesn’t support it.

Snap-ins — the older, rarely used version of modules — aren’t supported at all, which means the ActiveDirectory and DnsClient modules on Windows don’t work yet (the PowerShell and Windows Server teams are working on the AD module already). PowerShell Workflow is built on Windows Workflow Foundation, which needs .NET Framework, so that stays Windows only; “we asked the community and we didn’t hear a resounding ‘this is critical’,” Snover told us. The most popular workflow use is parallelism so Microsoft is planning to support concurrency for PowerShell Core, although it won’t come until PowerShell Core 6.1.

There’s a Windows PowerShell Compatibility Pack planned to bring back some Windows PowerShell features missing in PowerShell Core like aliases, WMI and perf cmdlest, as well as automatically setting the module path. WMI v1 cmdlets like Get-WmiObject don’t work anymore; they’ve all been replaced by WMI v2 cmdlets that often add extra features, but that’s still a change to make. Third parties will also need to update their modules — if those vendors are still in business.

There’s only a short list of compatible modules on the PowerShell Gallery so far (of the 30 three are deprecated and only a third aren’t from Microsoft) but when the PowerShell Standard 5.1 and Core 6.0 libraries come out later this month it will be easier for developers to create modules that are compatible with both.

Although Snover couldn’t comment on when various Microsoft teams would formally support PowerShell Core for Microsoft workloads, he noted that “Not only will Windows people be able to manage your stuff, so will Linux people; when a cross-platform value proposition is important, people [at Microsoft] are totally in. Azure is totally in and is going to be there pretty quickly.”

The Azure PowerShell team has several pre-release modules for using Azure Resource Manager with PowerShell Core in the PowerShell Galley, and the semantics for PowerShell cmdlets for Azure is being simplified. PowerShell Core support is “on the roadmap” for Azure Automation, but don’t expect to see it until a critical mass of cmdlets has been tested or updated.

In the long run, the change brings big advantages of course. “The big challenge right now is that cmdlets need to be supported in both frameworks; if you link against the .NET Standard library then you work both with full .NET and .NET Core. That means is cmdlets work with Windows PowerShell and PowerShell Core on Windows, and If you take that code onto Linux they run there as well, so we need to get the full community to support .NET Standard.”

Microsoft’s Desired State Configuration is getting the same deep refresh as PowerShell, although it’s not as far along. “Desired State Configuration has always been cross-platform from the beginning, it’s been available on Windows and Linux, on the agent side, but in reality, we have one code base Linux and another for Windows and we’re going to unify that codebase,” Snover said.

“We are rolling out Desired State Configuration widely through our Azure estate. If you look at the Azure portal, you’ll see we have configuration management there and we’re going to make that ubiquitous throughout Azure. And as we do that we’re picking up a new set of requirements and we’ll be coming out with a new design. It will be a hierarchal Desired State Configuration tool; in other words, there will be a well-defined interface that will be able to talk first to a native code agent that will be like the bootstrap agent and so a bunch of things. From there, it will be able to bootstrap into a .NET Core-based agent and from there a full managed code agent.”

The Just Enough Admin (JEA) PowerShell security controls work on Windows in PowerShell Core and will also be coming to Linux, Snover confirmed. “There’s a number of things we need to add to the Linux side but that’s our intent. In fact, wherever Linux allows something to be done, we’ll try and do it with PowerShell Core.”

PowerShell is shaping up to be another Microsoft open source success, with half of the changes in the new release coming from the PowerShell community. That’s where to look for the direction of PowerShell he told us. “We are now a community-based project and what that means it is completely open. We have a set of projects and priorities in our GitHub repo; there’s no second set of books, those are our books and the community gets to say ‘we think you’re wrong on this’. It’s a full open kimono. The community gets to take this wherever they need to take it.”

Microsoft is a sponsor of The New Stack.

Feature image via Pixabay.

The post Microsoft’s PowerShell Core Offers Cross-Platform Automation appeared first on The New Stack.

Moving away from monolithic apps to microservices, containers and REST APIs means that communications and message queues within applications becomes even more important, but traditional polling and pub-sub approaches don’t scale well to handle that. Whether it’s IoT or automated operations or serverless functions, event-driven computing is becoming foundational, and the Azure Event Grid service was created to fill this gap for event communications, Corey Sanders, Microsoft’s head of product for Azure Compute told the New Stack.

“As apps get more complex, as they take on more components, as they span multiple clouds and span hybrid, the requirements to communicate between them have become increasingly important. The previous way of doing this with polling mechanisms and listeners and so on is incredibly cumbersome and costly, so clearly an event-based solution is the way to go,” Sanders said.

In the longer term, Event Grid might move beyond cloud and become the basis of Azure Stack event handling, especially as the Azure Functions is already available there. For example, many IoT systems need to work for remote and low-bandwidth locations, using edge computing. “The primary scenario we see for Azure Stack is that latency-sensitive edge scenario; it’s the fastest growing usage case we see today with the solutions that focused on compute and storage and so on,” Sanders said.

Publish and Push

Event Grid has been in preview since the middle of last year, offering the primary way to handle events on Azure and beyond; now it’s generally available, and in more regions (initially Europe and Asia with more regions coming soon), new enterprise features and more developer tools.

Azure Event Grid service integrations.

Events are delivered by push semantics, so apps don’t have to continuously poll to look for changes. Event Grids supports events from key Azure services like storage, resource groups, Azure subscriptions and resource groups: the GA release adds General Purpose Storage and Azure IoT Hub as event publishers, as well as Azure Container Registry. That means that Event Grid shows up as an option in

But events don’t have to come from Azure services; they can be anything from tweets to IoT device updates, cloud provisioning, storage blob creation, new users being added to an HR system or products on a manufacturing line.

Finnish industrial processing service provider Outotec has been using the preview of Event Grid to rearchitect its hybrid integration platform, Sanders explained. “They have a bunch of plants and processing locations and manufacturing lines and they’re using [Event Grid] to take this central funnel of events happening from all of those locations, to be able to react to them on a service in Azure. It creates this opportunity for them to go rethink how the service is built without having to deal that any of underlying infrastructure.”

In and Out of Azure

Event handlers subscribe to topics (the endpoint where publishers send events); handlers also use subscriptions to filter incoming events, by type or publish path. Fan-out subscribes multiple endpoints to the same event, so it can be consumed in different ways.

Now that the service has launched, delivery SLAs are what Sanders called ‘enterprise focused’; “four nines SLA and a 24 hour retry period for any event inside the grid.”

Setting up an event subscription in the Azure portal, with prefix and suffix filters.

The native Azure event handlers include serverless options like Azure Functions, Logic Apps and the “no-code” Microsoft Flow service, as well as Azure Automation; Event Hubs is a new handler since preview and webhooks handle sending events to custom code and third-party services beyond Azure. “It’s not restricted to Azure cloud,” Sanders pointed out. “The ability to come in and say ‘I’m going to run custom code and I’m going to run it anywhere I want and take advantage of Event Grid’ we were excited about, because we felt it opened up opportunities. It has been a very common scenario where customers come in and use one side of the Azure platform with Event Grid and use custom on the other side. Sometimes the input is custom and the output is Azure Functions; sometimes the input is Azure Storage and the output is some custom event handler — or another cloud service or an on-prem system. The handling and firing of the events are equivalent, whether it’s an internal or external source — it doesn’t matter; Event Grid treats them the same.”

Many Event Grid customers are building IoT systems. “The two big scenarios that we see are IoT and operational workflow. Customer usage has been very focused on Functions usage, very focused on serverless usage, to enable IT solutions, all the way out to operations and management of your infrastructure. Some of most common scenarios where customers are deploying serverless solutions, it’s appending to existing applications it’s adding richness to  other applications, and Event Grid just makes that much easier.”

IoT systems tend to have a much higher volume of events, but many of the operational workloads running on Event Grid don’t go above the free tier of 100,000 monthly events. (The service handled an average of 300 million events a week during the preview.)

Writing custom code to use Event Grid is also getting simpler. There are new management and data plane SDKs for Python, Nodejs, Go and Ruby as well as the existing C#/.NET support, with more languages coming later in 2018.

“As we see more customers come in and look to build out more complex solutions using Event Grid they need to be able to do it in a  programmatic way,” Sanders told us. “The portal experience makes it very easy but when you’re doing it at a larger scale, say with a CI/CD pipeline or doing it as part an overall application rollout, being able to do programmatically is important and delivering whatever languages people want has been an on-going goal.”

It’s also easier to consume events; instead of needing the event subscriber to deserialize events, it can fetch the JSON schema for all supported event types from the event schema store.

Microsoft is a sponsor of The New Stack.

The post Now Live, Microsoft Azure Event Grid Ties Together Serverless Functions, Cloud Services appeared first on The New Stack.

As devices get smarter and more tools use machine learning, mobile processors have started adding dedicated cores to do that machine learning on the device, where the data and the application are, using a pre-trained model offloaded to the device, especially for time sensitive AI tasks like classification and tracking.

This approach is a tradeoff between the performance of using a more powerful processor to do machine learning in the cloud, against the latency of transferring data and decisions to and from the device, and the battery impact of running machine learning on a mobile device, Jem Davies, general manager of the ARM Machine Learning Group told The New Stack. It also offers more privacy for sensitive personal data; something that users and regulators are increasingly concerned about.

To make this more efficient, ARM is introducing specific processors designed “from the ground up” for machine learning, under a new initiative called Project Trillium. “We want high levels of machine learning performance with high levels of power efficiency. This is a design specifically targeting inference at the edge that gives us a massive uplift over traditional CPU and GPU architecture and over the sort of thing that’s being done on digital signal processors [DSPs],” he explained.

ARM ML includes a machine learning processor, an object detection processor and neural network software libraries “optimized to run on a range of ARM platforms up to the Cortex-A and Mali GPU.”

The object detection processor detects not just people but the direction they’re facing in, the trajectory if they’re moving (for tracking across different frames of video) and gesture and poses, in full HD at 60 frames per second; it’s the second generation of the technology already available in the Hive Camera. It can also be used to pre-process video going to the new Machine Learning processor and isolate areas of interest like faces for face recognition or hands for gesture control.

The ML processor isn’t restricted to image processing; it can run any machine learning workload that relies on inference — so it’s designed to take data models trained offline and run them in real time, whether that’s keyword detection for a smart speaker, pattern training in a smart thermostat, voice recognition, autonomous driving or data center monitoring.

ARM promises the 4.6 trillion ops per second mobile processors already deliver, with a further 2-4 times improvement in performance because the processor is designed specifically for machine learning, but on a power budget of 3 trillion ops per Watt; a combination Davies claims is “considerably in excess of anything else in the market.” To achieve that, the processor is optimized for the reduced precision integer and matrix multiplier accumulator arithmetic used in machine learning inference workloads, with special units for the operations in convolutional neural networks. It also uses what Davies calls an ‘intelligent memory system’ instead of a traditional fetch and decode memory architecture which uses more power.

“What we need to do is reach those level of power efficiency and higher performance using special optimized units for special functions but we also have to be clever about what data we load and use. A traditional architecture — CPU, GPU or DSP — is going to involve a lot of intermediate results storing and loading from memory, so we have produced a new form of operation that avoids that step. This is designed around the intelligent memory system that avoids temporary stores of intermediate results. We try to never reload a piece of data; if you have to reload an intermediate result you’ve failed. Once it’s loaded we try to use it as much as possible before storing it.”

Simplifying Machine Learning on Multiple Devices

ARM has also released ARM NN, a set of software optimized for the new processors that includes intermediate libraries to support a handful of the common machine learning frameworks (although not as wide a range as is supported by Qualcomm’s Neural Processing Engine SDK). “We augment the neural network frameworks that developers are comfortable and experienced with,” Davies told us.

“There are perhaps 20 frameworks in use today; the most common are TensorFlow, Caffe, Caffe2, MXNet and the Android NN API and two of those account for 80 percent of the instances.” He predicted some “coalescing and merging” (although ARM isn’t yet supporting the Open Neural Network Exchange (ONNX), the format that allows developers to change easily between frameworks). “We will support those partners who want the frameworks they want,” he promised, so the list may expand.

Some smartphone makers have already created custom “neural processing units” for their devices, like Google’s Pixel Visual Core, the Huawei NPU in the Kirin 970 chipset, and the iPhone’s Neural Engine (one of the custom signal processors in the A11 Bionic SoC, which performs tracking and object detection). Third-party apps can use those but most device manufacturers won’t have the expertise to design their own NPUs, and having an ML core that’s part of the platform means developers don’t have to adapt smart applications to the different NPU on each device.

“I like ARM’s approach as developers can write to one API and the workload will use the resources at its disposal, whether it be ARM CPU, GPU or discrete ML accelerator,” Patrick Moorhead, president of Moor Insights & Strategy told us.

Davies also suggested that this approach will let developers run machine learning workloads on classes of devices that haven’t previously had the capability to support them, meaning that they might be dealing with a much wider range of hardware, “but they don’t need to understand the complexity of differing hardware underneath.” We’ll see these processors first in premium smartphones with 7nm processor technology, but they’ll be added to more entry-level and mass-market processors in time.

Gartner is predicting over a billion units of smart surveillance cameras a year in ten years’ time, and machine learning will likely run on IoT devices as much as on smartphones. “This is a scalable architecture that over time will enable so many different segments down to always-on devices and up into cars and servers,” he said.

“We’re not looking at machine learning as a unique category; it will be native to everything that computers do,” Davies told us. “Wherever possible, machine learning will be done at the edge, on the device.”

Feature image: ARM.

The post ARM’s Project Trillium Brings Dedicated Machine Learning Hardware to Smart Devices appeared first on The New Stack.

In this interview recorded at the Open Source Leadership Summit, Azure CTO Mark Russinovich and The New Stack’s founder Alex Williams discussed how Microsoft builds on and contributes to open source for Azure’s artificial intelligence (AI) and machine learning. As Russinovich — who had just given a keynote suggesting that AI owes its current strength to the combination of open source and the cloud — explained, “Fundamentally a lot of AI, machine learning and analytics is built on top of open source and it’s a key part of our strategy to build with and use that open source, as well as to contribute back and to add to open source.”

Examples of that range from Microsoft contributing its own enhancements and fixes to existing projects like YARN (which is used in Azure Data Lake Analytics), to supporting the R open source community, to working with Facebook and AWS on the ONNX project to exchange models between Caffe, MXNet and CNTK, Microsoft’s own deep learning framework — which is also open source.

“CNTK is our own intellectual property; it’s a differentiated convolutional neural network framework… that we developed internally for Bring. We use that internally for a lot of our cognitive APIs… and we contributed that to [the] open source [community],” Russinovich explained.

The way Azure brings those open source components to customers is usually as part of a service, like the Cognitive Services APIs or the way R is integrated with SQL Server and Azure SQL DB so you can do your machine learning where your data is, he said. “Either we’re making it available for use on our platform or powering our platform and providing services to customers [with it].”

Increasingly, customers put those pieces together as part of a pipeline to create machine learning-driven apps, which is a good fit for new application-development practices like serverless computing. In the keynote, Russinovich showed a new app called DiagnostiCX that uses machine learning to help doctors who don’t have access to a trained radiologist interpret chest x-rays for signs of pneumonia — one of the biggest killers of children around the world.

“It works as a pipeline; a new x-ray comes in that kicks off a workflow, which will go do analytics on it, that will then go and do scoring, which will result in that score triggering some workflow like a producing report that the doctor takes an action on in the application they get. Everything is basically data-driven, which corresponds to events that are triggering the production and the processing of that data.”

Other open source technologies like containers are also useful for putting machine learning apps into production. Azure services like the Machine Learning Workbench create data models that can run both on the Azure Container Service or on IoT devices at the edge. “On a factory floor with anomalous object detection for finding faulty parts that’s done with image recognition where you don’t want to send all the images of the good parts up to the cloud for additional processing and classification to… figure out what’s causing the problems in the factory assembly line; you want to just take the anomalous ones and send them up. So, if you put the image detection right on the edge you save all that bandwidth.”

Azure also has tools like IoT Hub for bringing that all that together into a functional and manageable IoT application that can be provisioned remotely, with tools for both developers and the ops team, Russinovich pointed out. “What we’re looking at is let’s create an application framework and infrastructure to support the most efficient way to manage an entire application to deployment to operations at scale, regardless of the topology of it.” It’s that combination of cloud and open source that have unlocked the power of machine learning for developers, he maintains.

Microsoft is a sponsor of The New Stack.

The post Azure CTO: Open Source Is Key to Machine Learning in the Cloud, or on the Edge appeared first on The New Stack.

How good are computers at reading and understanding documents?

When Microsoft announced recently that it had matched human performance on the Stanford Question Answering Dataset (SQuAD), it was an important milestone in machine reading comprehension (MRC). It was closely followed by Alibaba achieving a similar score and beaten a few days later by a system from the Harbin Institute of Technology and Chinese AI company iFLYTEK.

It’s the same kind of regular improvements we’ve seen in speech and image recognition, and as usual, developers can expect to see some of these techniques packaged up in open source tools and commercial APIs for them to use.

Microsoft hasn’t published the details of the R-NET+ system, but an earlier version called R-NET previously achieved a record score on SQuAD. This was a gated attention-based recurrent network with a self-matching stage. The recurrent neural network uses word embeddings to represent the questions and the documents that provide answers to them (which picks out the phrases that the neural network to should pay attention to), with a gating mechanism that scores different sections of the document to emphasize the passages that are relevant to the answer and ignore passages that aren’t relevant.

Recurrent networks use directed connections between nodes as memory inside the network, but in practice, they only store a limited amount of context. That means one possible answer being processed by the network might not include all the sections of the document that contain clues to the answer. The self-matching layer uses the neural network to match each word in the source document against the rest of the document as well as against the question, to make sure all the available information gets used to predict which words in the document are the best answer.

Reading or Understanding?

But the announcement also generated something of a backlash because the term “reading comprehension” sounds a lot like “reading a document and understanding it as well as a human.” That might be the long-term aim of the field, but it’s not what the SQuAD dataset measures. Reading comprehension has very broad uses in search and beyond, but right now it’s much more like the tests you may remember from school, where you had to read a passage of text and then answer questions using just the information it contains.

SQuAD has over 100,000 questions, all of which are answered by a section of text in one of 536 Wikipedia articles; the questions were created by crowd workers on Mechanical Turk who were asked to come up with as many questions as they could, based on a short snippet of an article. It’s a big improvement on previous datasets which were much smaller, or took a simpler approach of multiple choice or fill in the blank, and it does require logical reasoning to infer answers from the source passages (like connecting a name in one sentence to a pronoun in the next sentence), but it’s still not reflective of the kind of open-ended questions users ask search engines and intelligent agents.

SQuAD questions don’t have abbreviations or spelling mistakes in, either, and the MRC system doesn’t have to look at multiple documents or even multiple passages in a document to come up with an answer, so they’re not synthesizing information or rewriting it to make it match the question. The source documents on Wikipedia are also clean, high-quality articles; not all the documents systems will need to read will be that clear. And there’s always an answer in the SQuAD text, so models can use tricks like picking the only date in a passage if the question starts with “when.” That means it doesn’t test whether a model can determine whether there is enough information to answer a particular question.

SQuAD is a dataset of pairs of questions and answers and the snippet of Wikipedia text they come from; this example is why Alibaba’s chief scientist for natural-language processing Luo Is claimed that “Objective questions such as ‘what causes rain’ can now be answered with high accuracy by machines.”

As Adam Trischler, senior research scientist in the Maluuba team, explained to us, “SQuAD cover a range of topics including science, politics and medical information, which is good coverage, but the language is very distinct from what you’d find in a novel or a news article, so models trained on SQuAD have some limitations in generalizing to, say, reading the news.”

News search is a particularly interesting area, he said. “One of the places we see question answering and generation applying really nicely is in more transient high-volume information sources like news. There’s a ton of knowledge in Wikipedia, but it’s more static. News is changing every day so a machine comprehension reading question and answer approach can have a lot of impact because it can sift through the new information rapidly.”

The Maluuba team also released the Frames dataset based on conversations between users and travel experts with details of hotels, flights and destinations, as a way of testing intelligent agents that can answer more complex questions as part of on-going conversations where the goal might change based on the information they supply. Effective MRC systems will need to work with a wide range of data sources and handle questions that arrive in different forms, so multiple datasets are key to improving comprehension models.

Learning to Ask Questions

As usual, having large datasets to train and test with is helping researchers improve the performance of reading comprehension models — but creating those data sets is slow and expensive, making it harder to use machine reading for topics where there isn’t a good test data set to learn from.

“Objective questions such as ‘what causes rain’ can now be answered with high accuracy by machines.”– Alibaba’s Luo Is

Another team at Microsoft Research is using transfer learning to build MRC systems for domains like a new disease, where there are plenty of source documents, but we don’t have existing, manually labeled datasets of questions and answers to train on. SynNet is a two-stage synthesis network that first learns what interesting information in a document looks like — key facts, named entities and semantic concepts that are frequently used — synthesizes those into answers and then learns to generate natural language questions that the “interesting” information could answer, using bi-directional Long Short-Term Memory Networks. Once trained, SynNet can be applied to a new domain where it can generate both questions and answers that can be used to train an MRC system. Training SynNet on SQuAD then using it with NewsQA gives almost as good results as from a system that was trained specifically on NewsQA.

The Maluuba team that Microsoft acquired last year created the NewsQA dataset with 120,000 Q&A pairs using CNN articles from the DeepMind Q&A dataset with questions and answers written by different people and validated by still others. Again, not all questions have answers, so the system has to recognize when it doesn’t have enough information; those that do require reasoning to answer; that includes synthesizing information from multiple sentences, recognizing answers that paraphrase the question using synonyms and using general knowledge to infer answers from incomplete information or related concepts.

As usual, having large datasets to train and test with is helping researchers improve the performance of reading comprehension models — but creating those data sets is slow and expensive, making it harder to use machine reading for topics where there isn’t a good test data set to learn from.

Interesting facts and synthesized questions and answers from SynNet.

The Maluuba team is also working on teaching systems to ask better, more informative questions, using reinforcement learning with rewards focused on what makes the question better — like being fluently phrased. A system that can ask good questions requires deeper reasoning ability because it has to both understand the source document and be able to generate natural language questions. It also has to know when the answer to a question can’t be answered fully by a document and it needs to ask more questions or look for more sources, Trischler told us.

“Sometimes the information you’re looking for isn’t in the document you have in front of you; your knowledge base is incomplete, and information is missing from it. We want to address the question of when the available knowledge is insufficient: how do you improve or add to it? You explore the limits of what you know and what you don’t know, what you understand and what needs further clarification. As people, if there’s something we don’t know one of the first things we do is ask a question: asking questions is this fundamental behavior we undertake to fill in the gaps and add to knowledge. We envision the same thing happening with literate machines,” Trischler said.

That will also mean MRC systems working across multiple documents. “If I’m an agent and I get a question from a user and maybe I find a source of information, but as I read it — as a machine — maybe something is still unclear, so I want to ask additional questions and maybe that directs me to another document I can reference against to answer more complicated questions.”

Literate Machines

Machine reading is a harder problem than recognition tasks like image recognition because of the ambiguity and complexity in language. MRC systems have to understand synonyms and context (tea and coffee are both hot drinks, Java can be a coffee growing region or a programming language, a cappuccino and a cup of coffee mentioned in the same document might be the same thing).

That means using background information that may not be in the source document, as well as synthesizing information from multiple phrases (which means parsing sentence structure and connecting nouns, pronouns and verbs across sentences) and summarizing it into an answer that matches the question.

But if the answer to a question like “why does it rain” includes clouds and gravity, the system doesn’t have to be able to explain the theory of gravity unless that’s also mentioned in the document. Reading comprehension is a “term of art” and doesn’t imply understanding in the human sense, any more than sentiment detection algorithms “understand” happiness and sadness.

The most obvious use of MRC is in search; instead of a link to a document or web page that has the answer to a query somewhere on the page, it can give an immediate answer, synthesized from the source. It also improves search for narrow and specific domains where there may not be a lot of the data that search algorithms usually depend on to give useful results.

But could also help with fact checking (are the statistics or political commitments quoted in an article backed up by the documents quoted as evidence), summarizing scientific papers (which of two methods covered gives the best results for a given task, which drug has the best results for patients with these characteristics), product recommendations (which product gets only positive reviews), diagnosis (which diseases have these symptoms, what treatments are recommended in this situation), extracting troubleshooting information from a manual or answering questions about how tax codes apply to a specific situation.

The question and answer format is a good match for an intelligent assistant like Cortana, and it’s one of the reasons that Microsoft acquired Maluuba. Announcing the news Microsoft executive vice president for artificial intelligence Harry Shum talked about an AI agent that would use machine comprehension to answer complex questions about, say, how tax law applied to a company to work like “a secretary with an intuitive knowledge of your company’s inner workings and deals. The agent would be able to answer your question in a company security-compliant manner by having a deeper understanding of the contents of your organization’s documents and emails, instead of simply retrieving a document by keyword matching, which happens today.”

To deliver that Maluuba is trying to create what product manager Rahul Mehrotra described to us as “literate machines”; Machines that can think, reason and communicate like humans, machines that can read text, understand text and then learn how to communicate, whether it’s written or orally.

“Our goal was to solve the problem of finding information. Today we use keywords and you have to read all this information to really find what you’re looking for. Our original idea was to build a literate machine that can answer any question a user has. [The next step of reading comprehension is] how can we move away from multiple choice where the answers are given, to something more extractive where the system can read a news article and extract the exact answer where multiple options aren’t available for it to look at.”

Conversational exchanges with an agent could help to narrow down search results, Trischler suggested. “A search for an image of trees will have a huge pool of returned results. You may be looking for something more specific. The algorithm has no idea what that is but if it has reasoning capacity, it can look at the images and divide them into classes and ask you questions to disambiguate. ‘Are you looking for trees in summer with green leaves or in autumn with nice red leaves?’ That could help people refine searches or just learn more about you to help personalize your tools.”

Microsoft is a sponsor of The New Stack

Feature image via Pixabay.

The post How Good Is Machine Learning at Understanding Text? appeared first on The New Stack.

Edge devices are designed to live at the edge of the network, often on a slow or occasional connection, and increasingly they’re used for much more than monitoring. Microsoft has created a managed service, called Azure IoT Edge, to put analytics and intelligence on those devices using containers. Think of it as a way to move cloud workloads like analytics, machine learning, SQL Server databases and even serverless functions onto devices that are centrally deployed and managed, even if they’re only occasionally connected.

As announced at the recent Build conference, those workloads now include Microsoft’s Cognitive Services; the first one available is Custom Vision, which uses the same ResNet-50 deep learning network that Microsoft Bing uses for image recognition, but with the final layer replaced by one trained against your specific images to do image classification and object recognition (finding and labeling the location of known objects in the image with a bounding box).

For object recognition, you can use the Custom Vision portal to find all the objects that have been recognized and tag them specifically (including adjusting the bounding boxes to delineate objects more precisely to help with the training). Upload as few as 50 labeled images per class as the training set and in less than a minute you have a trained model that you can deploy as a module to an Azure IoT Edge device.

Keep feeding the results of the image recognition back in (or add more training images) and you get a better model that you can update devices with.

What you get in the export is a ‘compact’ version of the model that runs on more constrained mobile devices and you need to select a compact domain to train the model in before exporting (currently the options are general, landmarks and retail, which isn’t quite as many domain options as if you call the model as an API). The tradeoff is that it may not be as accurate as the same data model running in the cloud where there are more resources, but it can run right where you need the recognition done, even if there’s no network connection.

The IoT Edge export from the Custom Vision service is a DockerFile, a TensorFlow model and service code that can run in a Windows or Linux container (on X64 or ARM). The Azure IoT Edge runtime is open source.

Courtesy of Microsoft.

If you want to create your own data models instead of using Cognitive Services, you can package those as Docker containers to deploy to IoT Edge in the same way. To simplify packaging up models built with Azure Machine Learning, use the AI Toolkit for Azure IoT Edge.

And now there are more ways to deploy those containers onto edge devices managed by IoT Edge than just through the Azure portal.

To make it easier to manage IoT Edge deployments from inside a Kubernetes environment like Azure Kubernetes Service (the new, less confusing name for the fully managed Kubernetes container orchestration service on Azure), using familiar Kubernetes concepts like manifests, Microsoft has just published an open source IoT Edge Virtual Kubelet provider on GitHub that you can install in your Kubernetes cluster using Helm.

“This gives you a consistent interface where you can push an entire Kubernetes workload across cloud and edge,” — Sam George.

As Sam George, director of Azure IoT engineering, explained to The New Stack at Build, “In a Kubernetes cluster, Kubernetes nodes are typically bound to a VM; if I push containers to the cluster, they get placed inside a VM. A Virtual Kubelet works like a proxy. It can interpret the Kubernetes commands and do something else with [the container] beside placing it in a VM.”

So far, Virtual Kubelets have been about bursting out from a Kubernetes cluster to a cloud Kubernetes service like Azure Container Instances or Hyper to get an advantage like scale or speed or per-second billing without having to worry about managing agent nodes. The ACI Connector is a Virtual Kubelet. Here, though, the IoT Edge Virtual Kubelet provider is placing the containers into edge devices that then look as if they’re part of the cluster.

Host the provider in a Kubernetes cluster and it shows up as just another node, George explained. “When you push containers to the IoT Edge provider we interpret that push as an IoT Edge deployment; we go through our Azure IoT Hub service and push that container out to an edge device. The Kubernetes master thinks it’s just got a big Kubernetes cluster running in the cloud but in fact, some of the containers are now running on edge devices.”

The provider is still in development and not all of the integrations are finished, but it promises a lot of flexibility. The big advantage of IoT Edge is bringing cloud services consistently to edge devices, and using Kubernetes makes that apply to more than the Azure services that are integrated with IoT Edge. You can use it to bring data models to edge devices, but if there’s a new version of, say, the compression algorithm you use, you can manage to deploy that to the cloud and edge systems in the same way.

The IoT Edge provider can push to multiple edge devices, using tags and device selector query to target the deployment to the correct edge devices. IoT devices with the same software configuration will often be connected to IoT hubs in different regions for performance or resiliency; creating a virtual Kubernetes node for each of those IoT hubs means the same kubectl command can apply deployment manifests to devices with the same configuration even though they’re attached to different hubs.

It doesn’t matter if those IoT devices are on slow connections, as long as they’re connected, George said. “The part that doesn’t work yet is when there are times when the edge device is disconnected. The Virtual Kubelet works great where there is connectivity to the edge at all time because it’s reporting back two-way communication with the health of each container.”

Microsoft is a sponsor of The New Stack.

Feature image via Pixabay.

The post Kubernetes for Edge Computing: The Microsoft Azure Approach appeared first on The New Stack.