Question on validation of mining models

Hi, guys,

Thanks for your kind attention.

Just want to make things perfectly work and make the most of our fantastic SQL Server 2005 Data Mining Engine. Can any of you here give me some super advices on the validation of the mining models. As we always see, the 3 aspects of a mining model are: Score, Population correct, and Predict Probability. So the question is: How can we combine these three aspects to best judge the mining models by being able to tell which model is the best one? And to what extent can we really trust these mining models?

These are very important before we can actually bring the models into work to convince other people who have no ideas what are going on with these models. Yes, we just want to convince them with the results of these models and make the most of them and best help them getting the most from their business operations etc.

By the way please can you explain a bit details on each of these aspects? Thanks again.

I am looking forward to hearing from you shortly and thanks bunch for your help.

With best regards,

Yours sincerely,

Hi, Guys,

I am still waiting for your advices.

Thanks.

With best regards,

Yours sincerely,

|||

I heard about a book "Quality Measures in Data Mining" by Fabrice Guillet and Howard J. Hamilton. It may contain some super advices.

Gigi Ciubuc

www.sqlserver.ro

|||

Hi, Gigi Ciubuc,

Thanks for the imformation. Any other ideas from any other experts here as well?

Thanks.

With best regards,

Yours sincerely,

Question on PMML for data mining

Hi, all experts here,

Thank you very much for your kind attention.

I am having a question on the PMML for data mining. As we know PMML enables the deployment of data mining models over all the major data mining vendors' data mining platform. I want to make sure what features of data mining models are exchanged actually? Are only the contents of the mining models? Or all the relative features (from data source, data source view, atrtibutes etc. ) ? And can we reprocess the data mining models from the destination platform where the mining models are deployed different from where they are created?

I am looking foreward to hearing from you shortly and thank you very much.

With best regards,

Yours sincerely,

PMML exports only the content of the mining model.

Data Source and Data Source View information is not exported, because these are specific to Microsoft Analysis Services Data Mining. So, in order to reprocess a mining model imported using PMML, you must ensure that the columns in the data dictionary section of the PMML file are available in the mining structure.

Similarly, if importing to a different destination, you would need to ensure that the columns in the data dictionary section of the PMML file could be bound to available columns.

|||

Hi, Donald,

Thank you so much for your advices. It's been very helpful for me.

With best regards,

Yours sincerely,

Question on Microsoft Association Rules

I build a data mining model to predict what are the best studying methods for the student to pass the examinaton.

Create Mining Model StudentAssociation (
Student_No long key,
Gender text discrete predict,
PassOrFail text discrete predict,
StudyMethod table predict ( MethodName text key )
) Using Microsoft_Association_Rules ( Minimum_Support=0.02, Minimum_Probability=0.03 )

The mining table will contain all the methods that the students use, no matter their examination is passed or failed.
The value of PassOrFail will have either 'Pass' or 'Fail'.

According to the above model, can I query the best studying methods?
Or I should only train the model with the student who pass the examination, and ignore all the failed.

Thanks.
Joe.

I wouldn't use AR, try Decision Trees, Naive Bayes, or Neural Nets. Also, if you're trying to predict what study methods indicate pass or fail, you only need to make PassOrFail predictable, not the nested table.

You definately need positive and negative examples to predict the result. You could create a clustering model on only the passing students to see which types of study methods group together for passing students.

|||

As your suggestion to create a clustering model on only the passing students, then I suppose to find that student uses both "method A" and "method B" will have a highest chance to pass the examination.
However, "Method A" and "Method B" may also have the highest probability to fail if I study the failed students. Am I right?
Do I need to create two clustering models, one for passing and one for fail, in order to have a complete picture?
Or, is there any methods to accomplish this task?

Thanks Jamie.

|||

Actually, if you only use passing students, then you won't see that Method A and Method B have higher chances than any other method, since all students passed. What you will see is what methods are used together by passing students. For example, there may be a group of passing students who use methods A and B, another using methods C and D, and yet another using A and D.

You are right in that these methods could just as easily be used by failing students. You should also create a similar model for failing students. You may find that the clusters are the same, they may be different, they may be similar with different proportions. For example, you may have an A and B cluster in both models, but in the passing model you find 40% of the students in A and B and only 5% in the failing model.

Another option if you want to use clustering is to make the Pass/Fail column "Predict Only" - in this case the algorithm will cluster based on all other attributes ignoring pass/fail, and then apply statistics for pass/fail across the clusters that were created. This will show if pass/fail is independent of method groupings or not (you can use the cluster diagram for this). However, if you want specifically to predict pass/fail based on study methods, you are better off using Trees or Neural Nets.

|||

Your answer is very useful for me.

Thanks Jamie.

Question on data mining report performance optimization

Hi, all experts here,

Would any of you give me any ideas for how could we optimize the report on data mining models? (as we know, for the data mining report, we have to select the mining model and the case table)

Hope it is clear for your advices and help.

Thanks a lot in advance and I am looking forward to hearing from you shortly.

With best regards,

Yours sincerely,

What kind of report is this? An accuracy report, such as lift chart or scatter plot?

Also, what kind of algorithm is involved?

|||

Yes, accuracy report and lift chart report. All SQL Server 2005 built-in algorithms there.

Any suggestions?

With best regards,

Yours sincerely,

|||

Service Pack 2 added some performance improvements in generating accuracy reports for the Naive Bayes algorithm.

Besides this, there is not much you can do to improve the performance of these reports.

You could try to use the tip here: http://sqlserverdatamining.com/DMCommunity/TipsNTricks/4566.aspx and implement your own accuracy chart using reporting services

What kind of problems are you running into? Could you please let us know how many test cases are you using and how much time does it take to populate the report?

|||The performance of the lift chart/etc. is directly related to the number of cases you are testing. I.e. if you cut the number of cases in half, you will cut the time in half.|||

Hi, all,

Thanks a lot for your very kind advices and help.

With best regards,

Yours sincerely,

Question on Data Mining add-in for Office Visio 2007

Hi, all experts here,

Thank you for your kind attention.

Data mining add-in for Office Visio 2007 is great. But what I am concerning is-to end users (to technical users) how could we deliver the data mining models results in a better way (e.g. integrate with other possible clients ) to make it dynamically and easily undertood?

Hope my question is clear for your help.

With best regards,

Yours sincerely,

Not sure what you are looking for. We do have the thin client viewers for tree models and NB/Clustering which are interactive. Using Visio you can save as a web page which provides some interactivity|||

Hi, Jamie,

Thanks a lot for your help.

With best regards,

Yours sincerely,

Question on column mappings between mining structure and case table for lift chart

Hi, all experts here,

I am a bit confused for the model evaluation (lift chart), should we map all the columns for both the mining structure and the case table? I mean for those predictive models, we have a predict column, shouldnt we ignore the mapping of the predictive column between the mining structure and the case table? But it seemes we are not allowed to miss the predictive column mapping between the mining structure and the case table.

Why is that? Could any experts here give me some explanation on that?

Hope my question is clear for your help.

Thanks a lot and I am looking forward to hearing from you shortly.

With best regards,

Yours sincerely,

The mapping of the predicted column is required to compare the prediction with the actual test data. It is not used in the actual prediction|||

Hi, Bogdan,

Thanks a lot for your advices.

With best regards,

Yours sincerely,