How To Use SIGNATE Competition

Step 1: User Registration

(1) Click "会員登録 (Register)."

Enter the required information on the new registration screen.

(2) Receive a confirmation email.

After clicking "会員登録 (Register)" in (1), a confirmation email will be sent to you.

(3) Click on "ユーザー登録 (user registration)."

To complete the registration process, click on "ユーザー登録 (user registration)" in the confirmation email received in (2).

Step 2: Participate in competitions

(1) Select a competition from the list.

Select a competition that best matches your skills and interests from the list of competitions.

We offer a variety of competitions to suit every stage of development. These include competitions with prizes for winners, competitions for students only, and practice competitions for beginners with no deadlines.

(2) Confirmation of contents

Find out what the competition is all about.

"Competition Top Header"

It shows the competition's end date and time, the number of submissions and participants, the rewards, and whether participants are eligible for points or medals.

ポイント/メダル対象

(Points/Medals Eligible)

ポイント/メダルなし

(Points/Medals None)

"説明 (Description)"

It provides an overview of the competition, including information on compensation, the schedule, eligibility requirements, terms of participation, and how to participate in the forum.

概要 (Summary):
It includes the main purpose of the competition, as well as an outline of the event.

報酬 (Rewards):
If rewards are offered, the details are provided.

スケジュール (Schedule):
It includes competition start and deadline dates, team composition deadlines, dates for determining winners, and award ceremony dates.

参加資格 (Entry qualifications):
Eligibility requirements, if any, are detailed.

参加規約 (Terms and Conditions of Participation):
There are set rules and regulations that must be followed when participating in the competition. In particular, please note that cheating activities such as cracking, cheating, and spoofing are prohibited.

フォーラム (Forum):
If the competition is linked to Discord, you will find instructions on how to join the dedicated competition channel created on Discord. You will be able to exchange ideas and opinions with other participants in the competition to discover and learn new things.

"課題 (Theme)"

Information about the theme's contents, evaluation methods, participation rules, and information disclosure policies is displayed.

課題内容 (Theme):
It includes a detailed description of the theme, as well as the objectives on which you will be asked to work in order to develop the model.

評価方法 (Evaluation Method):
This section describes the metrics used for evaluation and how the winners are determined. Understanding the evaluation metrics in depth is the first step to proceed to the intermediate stage.

参加ルール (Rules):
This section contains the rules for using the system and data, handling information, and implementation conventions. Read the rules carefully before starting the analysis.

情報公開ポリシー (Disclosure policy):
This defines the scope of the findings, products, etc. (models and analysis results) obtained from the competition that can be disclosed. For example, if the analysis results are "open to the public," you can post an article share your solution on social networking sites or blogs.

"リーダーボード (Leaderboard)"

Participants' rankings and scores are displayed in real time. Aim for a higher rank while checking your own.

* The public leaderboard is a provisional ranking updated in real time during the competition.

* The private leaderboard is a ranking based on the final evaluation, which will be determined after the competition ends.

"ナレッジ (Knowledge)"

Tutorials and winners' reports are displayed. Use them as references to improve your skills.

チュートリアル (tutorials *):
Tutorials are available in jupyter notebook format, including procedures of downloading data, modeling with basic approaches, and submission.

入賞者レポート (winners' reports *)：
After the competition ends, the winners' solutions will be published in PDF format.

* Only some competitions.

"チーム (Team)"

Some competitions allow you to participate as a team. Invite a user to join your team by entering their name using the "招待する (invite)" button.

* The “チーム (team)” tab will appear after signing in and agreeing to the Terms of Participation for competitions where this feature is available.

* For competitions where the team function is not available, the only method of participation is individual participation.

"投稿ファイル (submission files)"

A list of the files you have submitted is displayed, and you can check their scores and notes.

* The "投稿ファイル (submission files)" tab will appear after you sign in and agree to the terms of participation.

(3) Agree to the competition terms and conditions

Clicking the "参加 (Join)" button will display the competition's terms and conditions.
After reviewing the contents, check "I agree to the terms" and click "OK."

The "投稿 (submit)" button will appear once you have joined the competition.
Submit your files in the format specified here after the analysis and model creation.

* If you do not agree to the terms and conditions, you will not be allowed to participate in that competition.

* Some competitions may require you to sign an NDA and provide identification.

(4) Download dataset

A list of data files will be displayed. After confirming the description of the data, download it by clicking the "ダウンロード (download)" button.

* The "データ (data)" tab will appear after you sign in and agree to the terms of participation.

* Data distributed in the competition may only be used for the competition in question. Use or redistribution outside of the competition is prohibited.

* For some competitions, SIGNATE CLI (the official API command line interface) can be used to download data and submit files.

* For some competitions, data cannot be downloaded to the local environment and a cloud analysis environment may be required.

Step 3: Build a Model

Explore data and create a best predictive model!

Things You Should Know

Confirm the Rules

Please read through the competition rules so that your model won't violate them.

Manage Analysis Results

It would be helpful to reproduce the result when you become a preliminary winner. As there's a chance to win due to disqualification as such, don't be discouraged with bad scores!

Avoid Overfitting!

Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts its performance on new data, which results in a drop in the final ranking.

Premise

Please keep in mind to create a practical model as each competition has its own goal such as achieving business challenges, solving social problems, and sharing results of studies.

Guidelines

(1) Explore dataset

Once you have downloaded dataset, get started with figuring out the characteristics of the data. It is important to confirm the number of records in data, category of data (continuous, discrete, string), percentage of missing data, distribution of data, and task type (regression, classification, multi-class, multi-label).

After figuring out the characteristics of the data, it's time to start exploratory data analysis. It is good to analyze to find such as the relationship between the variables, which variables are closely related with the objective variable, and what kind of features seems to be created to gain insight for prediction accuracy improvement, by using the methods like clustering, correlation analysis, and other various approaches.

(2) Build a model

Once the exploratory data analysis is done, let's move on to the next step: feature engineering and creating a model. Evaluate your prediction accuracy based on the data used to train your model to estimate how good prediction it gives. Then submit the prediction on the test set to SIGNATE to see how it actually performs.

Keep engineering features that are likely to contribute to your model improvement and submit the prediction to confirm whether it works or not.

Make good use of techniques such as cross validation, bootstrapping, feature selection, and hyper parameters tuning to avoid overfitting.

Don't forget to manage your analysis results so that you can reproduce them later.

Step 4: Automatic Scoring

How good is your model? See how it performs!

Evaluation and Submission

Keep improving model

It's good to see how good other competitors models are and compare with yours. As the competition end date approaches, more and more submissions are made and the competition tends to become hot and intense. Keep improving your model so you can maintain your ranking.

Submission limits per day

Every competition has a limit to the number of submissions per day. Once you have reached the limit of the day, give your brain and machine a break and let's wait for the limit getting reset at 0:00.

Final ranking is determined by the private leaderboard

The test set is split into a public and private part in the competition. During the competition, scores on the leaderboard are computed based solely on a fraction of the test set. We use the rest of the set to evaluate and determine the final ranking after the competition.
Leaderboard automatically switches to the final evaluation the moment competition ends, where we sometimes see the leaderboard ranking shaken up.

Guidelines

(1) Submit predictions on the test set which should be made in the specified format by your trained model.

The purpose of the competition for the participants is to produce the model that predicts best on the test set whose objective variable is disclosed.

* Trying to figure out the true value of objective variables on the test set is regarded as cheating which is prohibited. When a participant is found to have cheated in the competition, he or she will be disqualified.

(2) Automatic Scoring

Submission score is automatically computed based on the evaluation criteria to be reported on the leaderboard along with its ranking.

Scores reported on the leaderboard during the competition are temporary ones and after the competition, they will be switched into final evaluation ones.

* The leaderboard will be switched to display the final ranking once the competition is closed.

(3) Iterate Step 3 and 4 to improve your score.

Step 5: Competition Closing Process

Get your model verified to claim the prize!

Procedures to claim the prize

Model submission

Preliminary winners are required to provide their code and documentation. Please make sure that your model can reproduce the final ranking score and does not violate the competition rules.
The documentation should be well written and polished.

Model verification

Preliminary winners are required to submit source code and documentation of their model to receive the prize money. Please see the guidelines below for further details.

Transferring intellectual property rights

In order to receive the prize money, the intellectual property rights in your model are required to be transferred to the competition sponsor.

* This may not be necessary for some competitions.

Guidelines

(1) Receive a notification email

We will send a notification email to preliminary competition winners.

* We will notify in a week after determining their final ranking.

* Please check your spam folder if the email is not sent to your inbox.

(2) Submit source code and documentation of the model

Following our team's instruction, submit your model documentation by the deadline.

* We would appreciate it if you could provide us with well-arranged code and document as it helps us to proceed the verification process smoothly.

* Please make sure that your submitting model is what you used to generate the final evaluation result.

(3) Get your model verified

After we receive your model submission, we will confirm whether the model violates the competition rules and can reproduce its final evaluation result.

* Please note that our team will contact you when in unexpected situations during the model verifying process.

* To proceed the process smoothly, please provide your model documentation in as much detail as possible.

Here are examples of required information:

Solution
Suggestions obtained from analysis and modeling
A procedure to reproduce the result

(4) Proceed to accept the payment of the prize

Almost done! By concluding a contract of such as transferring rights, you will acquire the right to accept the payment of the prize.

* After entering into this agreement, in principle, the payment of the prize money will be completed within 2 months.