What is the easiest way to train a neural network?

-
We've been surveying our customers since forever, but recently we've started to become more and more hungry on data. As an invoicing service provider for micro entrepreneurs, Zervant's customers are of all varieties. One interesting data snippet we've found out is that roughly half of our customers are part-time entrepreneurs.

But which half is it?

As you can perhaps imagine a part-time entrepreneur's needs can differ greatly from those of a full-time one. To be able to cater for those needs, we should know which half they each belong to.

This will not be a post about how to train a neural network in R, all the steps and nuances of the process or some big revelation that turned the seemingly doomed project to success. This will be a lot shorter. In fact, once I had my data in the format I wanted, it took me 5 minutes to train a neural network and a boosted decision tree for my classifier. On Azure ML Studio. 5 minutes.

If it's a simple algorithm you need to try out and just don't feel like coding it up in R or Python, then have a look at the Azure ML Studio. From what I've used, it's the quickest way to have something up and running and see results. The workflow is implemented in a visual programming 'language' of sorts, like so:


If you don't have experience in Azure ML Studio, I would recommend having a look at this free course on edX. The course isn't too advanced even if you don't have experience in machine learning algorithms.

There are some downsides, too. Once you have trained an algorithm, it might not be very easy to get that algorithm out of Azure ML. Azure ML scales parameters automatically, so even if you manage to get e.g. a regression model out, you might still have trouble knowing how the data was scaled before training the model. But that one is still do-able with a regression model. With something like a neural network it gets impossible. Or a boosted decision tree.

Azure ML Studio wants you to publish your trained algorithm as a web service rather than taking it out of the portal and implementing it yourself. And, getting any substantial volume through that API will mean you need to pay for the service.

At least for me Azure ML Studio lowers the threshold to try out a model at a problem. By doing so, it enables you to easily evaluate a range of models and see how well they perform. This can help you decide more quickly the ones to go after, in case you wish to implement them in e.g. Python.

Comments

Post a Comment

Popular posts from this blog

How to access AWS S3 with pyspark locally using AWS profiles tutorial

-
At Zervant , we currently use databricks for our ETL processes, and it's quite great. However, there's been some difficulty in setting up scripts that work both locally and on the databricks cloud. Specifically, databricks uses their own prorpietary libraries to connect to AWS S3 based on AWS hadoop 2.7. That version does not support accessing using AWS profiles. Internally, we use SSO to create temporary credentials for an AWS profile that then assumes a role. Therefore, reading the ACCESS_ID and ACCESS_SECRET from the .credentials file is something we don't want to do. In order to accomplish this, we need to set two hadoop configurations to the Spark Context fs.s3a.aws.credentials.provider com.amazonaws.auth.profile.ProfileCredentialsProvider This is done by running this line of code: sc._jsc.hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.profile.ProfileCredentialsProvider") Note! You need to set your environment var...
Read more

Snowflake UPSERT operation (aka MERGE)

-
You want to insert data to a table, but if a corresponding row already exists (by some rule, e.g. unique key) you want to update that instead of adding a new row, keeping the dataset's unique requirements intact. That's an "UPDATE AND INSERT" operation, or UPSERT. Some SQL languages have native support for it.  PostgreSQL has UPSERT as native. Also MySQL supports the operation with INSERT and ON DUPLICATE KEY UPDATE. How do you do UPSERT on Snowflake? Here's how: Snowflake UPSERT i.e. MERGE operation Snowflake's UPSERT is called MERGE and it works just as conveniently. It just has a different name. Here's the simple usage: MERGE INTO workspace.destination_table d USING workspace.source_table s ON d.id = s.id AND d.val1 = s.val1 WHEN MATCHED THEN update SET d.val2 = s.val2, d.val3 = s.val3 WHEN NOT MATCHED THEN INSERT (id, val1, val2, val3) VALUES (s.id, s.val1, s.val2, s.val3); Here the destination_table and source_table are of similar form,...
Read more