Question on dealing with missing values for training models

Hi, all,

Just really wonder what is the good idea to deal with missing values? Should we leave the missing values there in the traning data set ? Or replace it with other values?

What I am really concerned is that if we simply replace those missing values with other values, then how will it really affect the correctness of the training models?

I am looking forward to hearing from you for the above issue and it will be really great if we have any kind of best practices of dealing with this issue.

Thanks.

With best regards,

Yours sincerely,

I have the same question, what's the best practice to replace NULL values in the source data? Should they be left as NULL or replaced with some predetermined default value?|||

For most algorithms, NULL and NULL substituted with a value do not mean the same thing. NULL is usually ignored whereas a substituted NULL is a real state value. The correct modeling approach depends on your data. If NULL has some information in your scenario (e.g. NULL for State means customer from a different country), it's a good idea to sutstitute NULL with a default value before training. If however, NULL means absense of data and has no information, it should be left as is.

Hope this helps.

|||

Hi, Shuvro,

Thanks a lot. It's is quite clear to me now to have a clear idea on dealing with the Null values.

With best regards,

Yours sincerely,

Question on dealing with missing values for training models

Hi, all,

Just really wonder what is the good idea to deal with missing values? Should we leave the missing values there in the traning data set ? Or replace it with other values?

What I am really concerned is that if we simply replace those missing values with other values, then how will it really affect the correctness of the training models?

I am looking forward to hearing from you for the above issue and it will be really great if we have any kind of best practices of dealing with this issue.

Thanks.

With best regards,

Yours sincerely,

I have the same question, what's the best practice to replace NULL values in the source data? Should they be left as NULL or replaced with some predetermined default value?|||

For most algorithms, NULL and NULL substituted with a value do not mean the same thing. NULL is usually ignored whereas a substituted NULL is a real state value. The correct modeling approach depends on your data. If NULL has some information in your scenario (e.g. NULL for State means customer from a different country), it's a good idea to sutstitute NULL with a default value before training. If however, NULL means absense of data and has no information, it should be left as is.

Hope this helps.

|||

Hi, Shuvro,

Thanks a lot. It's is quite clear to me now to have a clear idea on dealing with the Null values.

With best regards,

Yours sincerely,

Question on cube granularity again

Hi, all,

Again, I wonder is it a good idea to design the granularity level of the cube as the same as in its underlying relational database e.g the most detailed level as it is in its fact table and its dimension table.

I am looking forward to hearing from you for any advices.

Thanks a lot in advance.

With best regards,

Yours sincerely,

It really depends on your requirements. I would say you probably should make the granularities the same unless it's going to cause serious performance problems with the processing and/or querying of your cube - it just means you have all the data your users might want available. Changing the granularity of your cube once you've gone into production would also be a bit messy.

Chris

|||

Hi, Chris,

Thanks a lot for your advices. Very good point.

With best regards,

Yours sincerely,

Question on attributes selection for un-supervised algorithms and supervised algorithms

Hi, all,

Thanks for your kind attention.

Just wonder is there any good idea for us to select attributes for training models? Both for non-supervised algorithms like Association Rules and Clustering etc. and supervised algorithms like decision tree etc.

It will be much interesting to hear from you for any best practices and popular methods of dealing with this issue.

I am looking forward to hearing from you and thanks for your advices.

With best regards,

Yours sincerely,

Hi,

I assume that you are trying to select a subset of all your attributes to train the models. SQL Server Data Mining Algorithms have built in feature selection methods. For example, the Microsoft Decision Trees support the following attribute scoring methods: Entropy, Bayesian with K2 Prior and Bayesian Dirichlet Equivalent with Uniform Prior (which is used by default). When feature selection is necessary, the algorithm calculates the scores for each attribute and only train trees with selected features (with top scores, of course). Other algorithms have similar feature selection mechanism.

Thanks,

Question on attributes selection for un-supervised algorithms and supervised algorithms

Hi, all,

Thanks for your kind attention.

Just wonder is there any good idea for us to select attributes for training models? Both for non-supervised algorithms like Association Rules and Clustering etc. and supervised algorithms like decision tree etc.

It will be much interesting to hear from you for any best practices and popular methods of dealing with this issue.

I am looking forward to hearing from you and thanks for your advices.

With best regards,

Yours sincerely,

Hi,

I assume that you are trying to select a subset of all your attributes to train the models. SQL Server Data Mining Algorithms have built in feature selection methods. For example, the Microsoft Decision Trees support the following attribute scoring methods: Entropy, Bayesian with K2 Prior and Bayesian Dirichlet Equivalent with Uniform Prior (which is used by default). When feature selection is necessary, the algorithm calculates the scores for each attribute and only train trees with selected features (with top scores, of course). Other algorithms have similar feature selection mechanism.

Thanks,